2023-12-12T01:04:56.784 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-12T01:04:56.784 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682 --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} --verbose -- /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682/orig.config.yaml 2023-12-12T01:04:56.829 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-12T01:04:56.923 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682 branch: wip-yuri8-testing-2023-12-11-1101 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} email: yweinste@redhat.com first_in_suite: false job_id: '7487682' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-11-1101 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - must scrub before tier agent can activate sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri8-testing-2023-12-11-1101 sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 3273 sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2023-12-11-1101 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_21b8309530d48c7266e1db5c3ccbe10963f99737/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 21b8309530d48c7266e1db5c3ccbe10963f99737 targets: smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVq2NjzGXmkp7DPL6KCQEodOUt0BYVqMLmYCZjb6e6IgfDIVAh8hm4iNuD/s27e+4LQQyV2trss7Y2xwdQ5SXQ= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKSVFcPDqJ8XRrlSVZrbZ4L9mT6PaGtZIWXoh9syfiDttc9FDyL3ae9WqhdrXChGpVVKzRDQ0YbQxR2ud6RChBk= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAydphWbWOpSV35lZvkWf0ch0mK0jdjGCWtoDQF8KDj4dCCGDcuAa3aIo6bV+3d1rsFBXaybpRxep17kx6T1jEc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs - ceph-volume extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - rados: clients: - client.2 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-11_23:27:14 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-12T01:04:56.924 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_21b8309530d48c7266e1db5c3ccbe10963f99737/qa; will attempt to use it 2023-12-12T01:04:56.924 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_21b8309530d48c7266e1db5c3ccbe10963f99737/qa/tasks 2023-12-12T01:04:56.924 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-12T01:04:56.926 INFO:teuthology.task.internal:Saving configuration 2023-12-12T01:04:56.943 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-12T01:04:56.950 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T01:04:56.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:59:30.826640', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVq2NjzGXmkp7DPL6KCQEodOUt0BYVqMLmYCZjb6e6IgfDIVAh8hm4iNuD/s27e+4LQQyV2trss7Y2xwdQ5SXQ='} 2023-12-12T01:04:57.001 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:59:30.828387', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKSVFcPDqJ8XRrlSVZrbZ4L9mT6PaGtZIWXoh9syfiDttc9FDyL3ae9WqhdrXChGpVVKzRDQ0YbQxR2ud6RChBk='} 2023-12-12T01:04:57.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:59:30.824700', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAydphWbWOpSV35lZvkWf0ch0mK0jdjGCWtoDQF8KDj4dCCGDcuAa3aIo6bV+3d1rsFBXaybpRxep17kx6T1jEc='} 2023-12-12T01:04:57.021 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-12T01:04:57.027 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-12-12T01:04:57.027 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2023-12-12T01:04:57.027 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2023-12-12T01:04:57.027 INFO:teuthology.run_tasks:Running task console_log... 2023-12-12T01:04:57.134 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa8826a6e50>, signals=[15]) 2023-12-12T01:04:57.135 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-12T01:04:57.143 INFO:teuthology.task.internal:Opening connections... 2023-12-12T01:04:57.143 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2023-12-12T01:04:57.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:04:57.212 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-12-12T01:04:57.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:04:57.283 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2023-12-12T01:04:57.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:04:57.354 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-12T01:04:57.363 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2023-12-12T01:04:57.386 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2023-12-12T01:04:57.386 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2023-12-12T01:04:57.447 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2023-12-12T01:04:57.447 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8" 2023-12-12T01:04:57.447 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2023-12-12T01:04:57.447 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-12T01:04:57.448 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-12T01:04:57.449 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2023-12-12T01:04:57.471 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-12-12T01:04:57.494 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-12-12T01:04:57.495 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-12-12T01:04:57.556 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2023-12-12T01:04:57.556 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8" 2023-12-12T01:04:57.556 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2023-12-12T01:04:57.556 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2023-12-12T01:04:57.556 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-12T01:04:57.557 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-12T01:04:57.558 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-12T01:04:57.558 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-12-12T01:04:57.580 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2023-12-12T01:04:57.603 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2023-12-12T01:04:57.603 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2023-12-12T01:04:57.665 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-12T01:04:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2023-12-12T01:04:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-12T01:04:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2023-12-12T01:04:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-12T01:04:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-12T01:04:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-12T01:04:57.666 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2023-12-12T01:04:57.688 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-12T01:04:57.700 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-12T01:04:57.708 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-12T01:04:57.708 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2023-12-12T01:04:57.711 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-12-12T01:04:57.713 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2023-12-12T01:04:57.733 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-12T01:04:57.795 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-12T01:04:57.795 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2023-12-12T01:04:57.798 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-12-12T01:04:57.800 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2023-12-12T01:04:57.854 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-12T01:04:57.952 INFO:teuthology.run_tasks:Running task kernel... 2023-12-12T01:04:57.969 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-12T01:04:57.970 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-12T01:04:57.970 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-12T01:04:57.971 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-12T01:04:57.971 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-12T01:04:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:04:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:04:57.994 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2023-12-12T01:04:57.995 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-12-12T01:04:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:04:57.997 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2023-12-12T01:04:58.055 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-489.el8.x86_64 2023-12-12T01:04:58.056 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-489.el8.x86_64 2023-12-12T01:04:58.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2023-12-12T01:04:58.057 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-489.el8.x86_64 2023-12-12T01:04:58.057 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-489.el8.x86_64 2023-12-12T01:04:58.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2023-12-12T01:04:58.058 INFO:teuthology.orchestra.run.smithi100.stdout:4.18.0-489.el8.x86_64 2023-12-12T01:04:58.058 INFO:teuthology.task.kernel:Running kernel on smithi100: 4.18.0-489.el8.x86_64 2023-12-12T01:04:58.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2023-12-12T01:05:01.480 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - AppStream 38 MB/s | 35 MB 00:00 2023-12-12T01:05:01.635 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 8 - AppStream 35 MB/s | 35 MB 00:00 2023-12-12T01:05:01.660 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 37 MB/s | 35 MB 00:00 2023-12-12T01:05:08.913 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - BaseOS 51 MB/s | 56 MB 00:01 2023-12-12T01:05:09.019 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 52 MB/s | 56 MB 00:01 2023-12-12T01:05:09.369 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 8 - BaseOS 43 MB/s | 56 MB 00:01 2023-12-12T01:05:17.135 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 111 kB/s | 18 kB 00:00 2023-12-12T01:05:17.137 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras 60 kB/s | 18 kB 00:00 2023-12-12T01:05:17.477 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 43 kB/s | 6.9 kB 00:00 2023-12-12T01:05:17.479 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 8 - Extras common packages 42 kB/s | 6.9 kB 00:00 2023-12-12T01:05:17.676 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 8 - Extras 59 kB/s | 18 kB 00:00 2023-12-12T01:05:17.772 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-12-12T01:05:17.787 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-12-12T01:05:18.052 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 8 - Extras common packages 35 kB/s | 6.9 kB 00:00 2023-12-12T01:05:18.366 INFO:teuthology.orchestra.run.smithi100.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-12-12T01:05:19.229 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-12-12T01:05:20.323 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 16 MB 00:02 2023-12-12T01:05:23.021 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 515 kB/s | 24 kB 00:00 2023-12-12T01:05:24.164 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 426 kB/s | 24 kB 00:00 2023-12-12T01:05:24.791 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 16 MB 00:06 2023-12-12T01:05:26.649 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-12T01:05:26.649 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-12T01:05:26.936 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 4.18.0-527.el8 baseos 10 M 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 4.18.0-527.el8 baseos 43 M 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 4.18.0-527.el8 baseos 36 M 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2023-12-12T01:05:26.937 INFO:teuthology.orchestra.run.smithi100.stdout:Install 3 Packages 2023-12-12T01:05:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-12-12T01:05:26.938 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 90 M 2023-12-12T01:05:26.938 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 96 M 2023-12-12T01:05:26.938 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2023-12-12T01:05:27.575 INFO:teuthology.orchestra.run.smithi100.stdout:(1/3): kernel-4.18.0-527.el8.x86_64.rpm 22 MB/s | 10 MB 00:00 2023-12-12T01:05:27.774 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-12T01:05:27.774 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-12T01:05:28.043 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-12-12T01:05:28.044 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-12-12T01:05:28.044 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-12-12T01:05:28.044 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-12-12T01:05:28.044 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-12-12T01:05:28.044 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 4.18.0-527.el8 baseos 10 M 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 4.18.0-527.el8 baseos 43 M 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 4.18.0-527.el8 baseos 36 M 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:Install 3 Packages 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 90 M 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2023-12-12T01:05:28.045 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-12-12T01:05:28.106 INFO:teuthology.orchestra.run.smithi100.stdout:(2/3): kernel-modules-4.18.0-527.el8.x86_64.rpm 37 MB/s | 36 MB 00:00 2023-12-12T01:05:28.541 INFO:teuthology.orchestra.run.smithi137.stdout:(1/3): kernel-4.18.0-527.el8.x86_64.rpm 22 MB/s | 10 MB 00:00 2023-12-12T01:05:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:(3/3): kernel-core-4.18.0-527.el8.x86_64.rpm 30 MB/s | 43 MB 00:01 2023-12-12T01:05:28.581 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2023-12-12T01:05:28.582 INFO:teuthology.orchestra.run.smithi100.stdout:Total 55 MB/s | 90 MB 00:01 2023-12-12T01:05:28.613 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 416 kB/s | 24 kB 00:00 2023-12-12T01:05:29.108 INFO:teuthology.orchestra.run.smithi137.stdout:(2/3): kernel-modules-4.18.0-527.el8.x86_64.rpm 35 MB/s | 36 MB 00:01 2023-12-12T01:05:29.260 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2023-12-12T01:05:29.541 INFO:teuthology.orchestra.run.smithi137.stdout:(3/3): kernel-core-4.18.0-527.el8.x86_64.rpm 30 MB/s | 43 MB 00:01 2023-12-12T01:05:29.542 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-12-12T01:05:29.542 INFO:teuthology.orchestra.run.smithi137.stdout:Total 60 MB/s | 90 MB 00:01 2023-12-12T01:05:29.700 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2023-12-12T01:05:29.700 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2023-12-12T01:05:30.218 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-12-12T01:05:30.652 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-12-12T01:05:30.652 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-12-12T01:05:30.939 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2023-12-12T01:05:30.966 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2023-12-12T01:05:31.888 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-12-12T01:05:31.905 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-12-12T01:05:32.221 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-12T01:05:32.221 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-12T01:05:32.509 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-527.el8 baseos 10 M 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 4.18.0-527.el8 baseos 43 M 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 4.18.0-527.el8 baseos 36 M 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2023-12-12T01:05:32.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-12T01:05:32.511 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 90 M 2023-12-12T01:05:32.511 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2023-12-12T01:05:32.511 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-12-12T01:05:33.148 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): kernel-4.18.0-527.el8.x86_64.rpm 25 MB/s | 10 MB 00:00 2023-12-12T01:05:33.714 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): kernel-modules-4.18.0-527.el8.x86_64.rpm 37 MB/s | 36 MB 00:00 2023-12-12T01:05:33.773 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2023-12-12T01:05:34.173 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): kernel-core-4.18.0-527.el8.x86_64.rpm 30 MB/s | 43 MB 00:01 2023-12-12T01:05:34.173 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-12-12T01:05:34.173 INFO:teuthology.orchestra.run.smithi107.stdout:Total 54 MB/s | 90 MB 00:01 2023-12-12T01:05:34.315 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:05:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-12-12T01:05:34.852 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-12-12T01:05:35.277 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-12-12T01:05:35.277 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-12-12T01:05:35.309 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:05:36.515 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-12-12T01:05:36.535 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-12-12T01:05:37.881 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:05:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:05:38.916 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:05:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-12-12T01:05:39.326 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:05:39.910 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:05:41.700 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:05:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:05:42.754 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:05:43.106 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:05:43.565 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:05:43.917 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:05:47.342 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:05:47.621 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:03.903 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:03.927 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:04.278 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:04.278 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:06:04.278 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:06:05.178 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:05.204 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:05.355 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:05.355 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-12-12T01:06:05.355 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2023-12-12T01:06:05.355 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-4.18.0-527.el8.x86_64 kernel-core-4.18.0-527.el8.x86_64 2023-12-12T01:06:05.355 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-4.18.0-527.el8.x86_64 2023-12-12T01:06:05.356 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-12-12T01:06:05.356 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2023-12-12T01:06:05.498 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T01:06:05.572 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:05.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:06:05.573 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:06:05.714 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T01:06:05.714 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T01:06:05.714 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-527.el8.x86_64 2023-12-12T01:06:05.715 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-12T01:06:05.715 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-12T01:06:05.715 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-12T01:06:05.715 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2023-12-12T01:06:05.715 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T01:06:05.924 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T01:06:05.925 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T01:06:05.925 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2023-12-12T01:06:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2023-12-12T01:06:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2023-12-12T01:06:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.02 2023-12-12T01:06:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 129.el8 2023-12-12T01:06:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2023-12-12T01:06:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Group : System Environment/Base 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 9265586 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Relocations : (not relocatable) 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : CentOS BuildSystem 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2023-12-12T01:06:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-12-12T01:06:05.961 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2023-12-12T01:06:05.963 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-12T01:06:05.963 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 4.18.0-527.el8.x86_64 2023-12-12T01:06:05.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-12T01:06:06.345 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:06.345 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-12-12T01:06:06.346 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-12-12T01:06:06.346 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-4.18.0-527.el8.x86_64 kernel-core-4.18.0-527.el8.x86_64 2023-12-12T01:06:06.346 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-4.18.0-527.el8.x86_64 2023-12-12T01:06:06.346 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-12-12T01:06:06.346 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-12-12T01:06:06.512 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T01:06:06.727 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T01:06:06.727 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T01:06:06.727 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-527.el8.x86_64 2023-12-12T01:06:06.727 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-12T01:06:06.727 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-12T01:06:06.727 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-12-12T01:06:06.727 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2023-12-12T01:06:06.727 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T01:06:06.937 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T01:06:06.937 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T01:06:06.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2023-12-12T01:06:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2023-12-12T01:06:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2023-12-12T01:06:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.02 2023-12-12T01:06:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 129.el8 2023-12-12T01:06:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2023-12-12T01:06:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Group : System Environment/Base 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 9265586 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Relocations : (not relocatable) 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : CentOS BuildSystem 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-12-12T01:06:06.971 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-12T01:06:06.972 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-12T01:06:06.972 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-12T01:06:06.972 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2023-12-12T01:06:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-12-12T01:06:06.972 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2023-12-12T01:06:06.973 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-12T01:06:06.973 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 4.18.0-527.el8.x86_64 2023-12-12T01:06:06.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-12T01:06:07.417 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2023-12-12T01:06:08.389 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2023-12-12T01:06:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:09.601 INFO:teuthology.orchestra.run.smithi100.stderr:done 2023-12-12T01:06:09.603 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2023-12-12T01:06:09.644 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.1WEVJBbWs9 2023-12-12T01:06:09.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1WEVJBbWs9 2023-12-12T01:06:09.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.1WEVJBbWs9 2023-12-12T01:06:09.795 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:09.795 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-527.el8.x86_64 1/3 2023-12-12T01:06:09.795 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-4.18.0-527.el8.x86_64 2/3 2023-12-12T01:06:09.828 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.1WEVJBbWs9 is 6KB 2023-12-12T01:06:09.878 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.1WEVJBbWs9 2023-12-12T01:06:09.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2023-12-12T01:06:09.997 INFO:teuthology.orchestra.run.smithi100.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-12T01:06:09.998 INFO:teuthology.orchestra.run.smithi100.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-12T01:06:09.998 INFO:teuthology.orchestra.run.smithi100.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-12T01:06:09.998 INFO:teuthology.orchestra.run.smithi100.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64.conf 2023-12-12T01:06:09.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64 2023-12-12T01:06:10.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-4.18.0-527.el8.x86_64 3/3 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-527.el8.x86_64 kernel-core-4.18.0-527.el8.x86_64 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-4.18.0-527.el8.x86_64 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-12T01:06:10.590 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-12-12T01:06:10.661 INFO:teuthology.orchestra.run.smithi137.stderr:done 2023-12-12T01:06:10.662 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2023-12-12T01:06:10.701 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.RyJv9RK7S4 2023-12-12T01:06:10.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RyJv9RK7S4 2023-12-12T01:06:10.738 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T01:06:10.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.RyJv9RK7S4 2023-12-12T01:06:10.927 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.RyJv9RK7S4 is 6KB 2023-12-12T01:06:10.953 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T01:06:10.953 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T01:06:10.953 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-527.el8.x86_64 2023-12-12T01:06:10.953 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-12T01:06:10.953 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-12T01:06:10.954 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-12T01:06:10.954 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2023-12-12T01:06:10.954 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2023-12-12T01:06:10.975 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.RyJv9RK7S4 2023-12-12T01:06:10.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2023-12-12T01:06:11.077 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-12T01:06:11.078 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-12T01:06:11.078 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-12T01:06:11.078 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64.conf 2023-12-12T01:06:11.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64 2023-12-12T01:06:11.161 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-527.el8.x86_64 2023-12-12T01:06:11.162 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-527.el8.x86_64 2023-12-12T01:06:11.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2023-12-12T01:06:11.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.02 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 129.el8 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Group : System Environment/Base 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 9265586 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-12T01:06:11.195 INFO:teuthology.orchestra.run.smithi107.stdout:Relocations : (not relocatable) 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : CentOS BuildSystem 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-12-12T01:06:11.196 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2023-12-12T01:06:11.198 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-12T01:06:11.198 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 4.18.0-527.el8.x86_64 2023-12-12T01:06:11.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-12T01:06:12.975 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2023-12-12T01:06:15.128 INFO:teuthology.orchestra.run.smithi107.stderr:done 2023-12-12T01:06:15.129 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-12-12T01:06:15.161 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.3p4tOjTg8t 2023-12-12T01:06:15.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3p4tOjTg8t 2023-12-12T01:06:15.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.3p4tOjTg8t 2023-12-12T01:06:15.346 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.3p4tOjTg8t is 6KB 2023-12-12T01:06:15.397 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.3p4tOjTg8t 2023-12-12T01:06:15.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2023-12-12T01:06:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-12T01:06:15.485 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-12T01:06:15.486 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-12T01:06:15.486 INFO:teuthology.orchestra.run.smithi107.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64.conf 2023-12-12T01:06:15.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-527.el8.x86_64 2023-12-12T01:06:15.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2023-12-12T01:06:40.134 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-12T01:06:40.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-12-12T01:06:40.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:06:41.198 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-12-12T01:06:41.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-12-12T01:06:41.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:06:45.662 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-12T01:06:45.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-12T01:06:45.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:07:20.291 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-12-12T01:07:29.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-12T01:07:29.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:07:40.146 DEBUG:teuthology.orchestra.remote:timed out 2023-12-12T01:07:41.202 DEBUG:teuthology.orchestra.remote:timed out 2023-12-12T01:07:49.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-12-12T01:07:49.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:07:49.690 DEBUG:teuthology.orchestra.run.smithi100:> true 2023-12-12T01:07:50.012 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-12-12T01:07:50.012 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-527.el8.x86_64"... 2023-12-12T01:07:50.012 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2023-12-12T01:07:50.029 INFO:teuthology.orchestra.run.smithi100.stdout:4.18.0-527.el8.x86_64 2023-12-12T01:07:50.030 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-527.el8.x86_64 vs 4.18.0-527.el8.x86_64 2023-12-12T01:07:50.030 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-12T01:07:50.030 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-12T01:07:50.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-12-12T01:07:50.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:07:50.532 DEBUG:teuthology.orchestra.run.smithi137:> true 2023-12-12T01:07:50.834 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-12-12T01:07:50.834 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-527.el8.x86_64"... 2023-12-12T01:07:50.835 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2023-12-12T01:07:50.851 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-527.el8.x86_64 2023-12-12T01:07:50.852 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-527.el8.x86_64 vs 4.18.0-527.el8.x86_64 2023-12-12T01:07:50.852 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-12T01:07:50.852 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-12T01:07:51.031 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-12T01:07:51.031 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-12T01:07:51.100 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2023-12-12T01:07:51.121 DEBUG:teuthology.parallel:result is None 2023-12-12T01:07:51.853 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-12-12T01:07:51.853 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-12T01:07:51.929 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2023-12-12T01:07:51.941 DEBUG:teuthology.parallel:result is None 2023-12-12T01:08:01.140 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-12-12T01:08:01.394 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-12-12T01:08:01.394 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-527.el8.x86_64"... 2023-12-12T01:08:01.395 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-12-12T01:08:01.411 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-527.el8.x86_64 2023-12-12T01:08:01.411 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-527.el8.x86_64 vs 4.18.0-527.el8.x86_64 2023-12-12T01:08:01.411 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-12T01:08:01.412 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-12T01:08:02.412 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-12T01:08:02.412 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-12T01:08:02.482 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2023-12-12T01:08:02.497 DEBUG:teuthology.parallel:result is None 2023-12-12T01:08:02.497 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-12T01:08:02.505 INFO:teuthology.task.internal:Creating test directory... 2023-12-12T01:08:02.506 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-12T01:08:02.509 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-12T01:08:02.511 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-12T01:08:02.537 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-12T01:08:02.545 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-12T01:08:02.554 INFO:teuthology.task.internal:Creating archive directory... 2023-12-12T01:08:02.555 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-12T01:08:02.573 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-12T01:08:02.575 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-12T01:08:02.609 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-12T01:08:02.617 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-12T01:08:02.617 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-12T01:08:02.641 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-12T01:08:02.644 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-12T01:08:02.673 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:08:02.681 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:08:02.681 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:08:02.683 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:08:02.692 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:08:02.693 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:08:02.695 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-12T01:08:02.700 INFO:teuthology.task.internal:Configuring sudo... 2023-12-12T01:08:02.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-12T01:08:02.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-12T01:08:02.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-12T01:08:02.768 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-12T01:08:02.777 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-12T01:08:02.778 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-12T01:08:02.806 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-12T01:08:02.808 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-12T01:08:02.827 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:08:02.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:08:02.956 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:08:03.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:08:03.083 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-12-12T01:08:03.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-12T01:08:03.150 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:08:03.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:08:03.242 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:08:03.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:08:03.369 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-12-12T01:08:03.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-12T01:08:03.437 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:08:03.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:08:03.545 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:08:03.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:08:03.676 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-12-12T01:08:03.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-12T01:08:03.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2023-12-12T01:08:03.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-12-12T01:08:03.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2023-12-12T01:08:03.801 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T01:08:03.806 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T01:08:03.849 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T01:08:04.157 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-12T01:08:04.164 INFO:teuthology.task.internal:Starting timer... 2023-12-12T01:08:04.164 INFO:teuthology.run_tasks:Running task pcp... 2023-12-12T01:08:04.174 INFO:teuthology.run_tasks:Running task selinux... 2023-12-12T01:08:04.183 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-12T01:08:04.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2023-12-12T01:08:04.274 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2023-12-12T01:08:04.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2023-12-12T01:08:04.362 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2023-12-12T01:08:04.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2023-12-12T01:08:04.455 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2023-12-12T01:08:04.456 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-12T01:08:04.456 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2023-12-12T01:08:04.497 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2023-12-12T01:08:04.497 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2023-12-12T01:08:04.526 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2023-12-12T01:08:04.527 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2023-12-12T01:08:04.549 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2023-12-12T01:08:04.550 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2023-12-12T01:08:04.550 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T01:08:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:08:04.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T01:08:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:08:04.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T01:08:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:08:04.633 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-12T01:08:04.633 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2023-12-12T01:08:04.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2023-12-12T01:08:04.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2023-12-12T01:08:04.730 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-12T01:08:04.739 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-12T01:08:04.744 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-12T01:08:04.881 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-12T01:08:04.900 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-12T01:08:04.901 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com,smithi107.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-12T01:14:12.109 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682/ansible_failures.yaml 2023-12-12T01:14:12.111 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: INTERNAL SERVER ERROR', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: Internal Server Error', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-12T01:14:12.194 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=430b231412934625b724b7151e90f08f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: INTERNAL SERVER ERROR', 'Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: Internal Server Error', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-12T01:14:12.196 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-12T01:14:12.206 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-12T01:14:12.207 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-12T01:14:12.225 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-12T01:14:12.320 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-12T01:14:12.407 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-12T01:14:12.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T01:14:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:14:12.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T01:14:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:14:12.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-12T01:14:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:14:12.597 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-12T01:14:12.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-12T01:14:12.617 INFO:teuthology.task.internal:Duration was 368.453475 seconds 2023-12-12T01:14:12.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-12T01:14:12.631 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-12T01:14:12.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-12T01:14:12.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-12T01:14:12.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-12T01:14:12.682 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T01:14:12.683 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T01:14:12.684 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-12T01:14:12.999 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-12T01:14:12.999 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2023-12-12T01:14:12.999 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-12T01:14:13.025 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-12-12T01:14:13.025 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-12T01:14:13.049 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2023-12-12T01:14:13.049 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-12T01:14:13.072 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-12T01:14:13.072 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-12T01:14:13.074 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-12T01:14:13.092 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-12T01:14:13.148 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-12T01:14:13.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-12T01:14:13.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-12T01:14:13.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-12T01:14:13.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-12T01:14:13.257 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-12T01:14:13.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-12T01:14:13.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-12T01:14:13.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-12T01:14:13.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-12T01:14:13.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-12T01:14:13.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-12T01:14:13.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-12T01:14:13.372 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2023-12-12T01:14:13.376 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-12-12T01:14:13.403 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2023-12-12T01:14:13.426 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-12T01:14:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:14:13.453 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-12T01:14:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:14:13.470 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-12T01:14:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:14:13.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-12T01:14:13.498 INFO:teuthology.task.internal:Transferring archived files... 2023-12-12T01:14:13.500 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682/remote/smithi100 2023-12-12T01:14:13.501 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-12T01:14:13.556 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682/remote/smithi107 2023-12-12T01:14:13.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-12T01:14:13.603 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682/remote/smithi137 2023-12-12T01:14:13.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-12T01:14:13.682 INFO:teuthology.task.internal:Removing archive directory... 2023-12-12T01:14:13.683 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-12T01:14:13.685 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-12T01:14:13.687 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-12T01:14:13.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-12T01:14:13.729 INFO:teuthology.task.internal:Not uploading archives. 2023-12-12T01:14:13.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-12T01:14:13.743 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-12T01:14:13.743 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-12T01:14:13.745 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-12T01:14:13.748 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-12T01:14:13.760 INFO:teuthology.orchestra.run.smithi100.stdout: 525359 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 12 01:14 /home/ubuntu/cephtest 2023-12-12T01:14:13.763 INFO:teuthology.orchestra.run.smithi107.stdout: 525683 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 12 01:14 /home/ubuntu/cephtest 2023-12-12T01:14:13.764 INFO:teuthology.orchestra.run.smithi137.stdout: 525359 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 12 01:14 /home/ubuntu/cephtest 2023-12-12T01:14:13.766 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-12T01:14:13.775 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-12T01:14:13.801 INFO:teuthology.nuke:Checking targets against current locks 2023-12-12T01:14:13.832 DEBUG:teuthology.nuke:shortname: smithi100 2023-12-12T01:14:13.832 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T01:14:13.859 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:59:30.826640', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVq2NjzGXmkp7DPL6KCQEodOUt0BYVqMLmYCZjb6e6IgfDIVAh8hm4iNuD/s27e+4LQQyV2trss7Y2xwdQ5SXQ='} 2023-12-12T01:14:13.863 DEBUG:teuthology.nuke:shortname: smithi107 2023-12-12T01:14:13.863 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T01:14:13.888 DEBUG:teuthology.nuke:shortname: smithi137 2023-12-12T01:14:13.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T01:14:13.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:59:30.828387', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKSVFcPDqJ8XRrlSVZrbZ4L9mT6PaGtZIWXoh9syfiDttc9FDyL3ae9WqhdrXChGpVVKzRDQ0YbQxR2ud6RChBk='} 2023-12-12T01:14:13.916 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-11_23:27:14-rados-wip-yuri8-testing-2023-12-11-1101-distro-default-smithi/7487682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 00:59:30.824700', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAydphWbWOpSV35lZvkWf0ch0mK0jdjGCWtoDQF8KDj4dCCGDcuAa3aIo6bV+3d1rsFBXaybpRxep17kx6T1jEc='} 2023-12-12T01:14:13.927 INFO:teuthology.orchestra.console.smithi100:Power off 2023-12-12T01:14:13.928 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-12T01:14:13.953 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off 2023-12-12T01:14:13.954 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:13.974 INFO:teuthology.orchestra.console.smithi107:Power off 2023-12-12T01:14:13.974 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-12T01:14:13.988 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2023-12-12T01:14:13.999 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-12-12T01:14:14.000 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:14.021 INFO:teuthology.orchestra.console.smithi137:Power off 2023-12-12T01:14:14.022 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-12T01:14:14.032 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-12-12T01:14:14.046 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2023-12-12T01:14:14.046 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:14.071 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-12-12T01:14:17.989 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:18.034 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:18.072 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:18.100 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2023-12-12T01:14:18.144 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-12-12T01:14:18.183 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-12-12T01:14:22.102 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:22.145 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:22.184 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:22.213 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2023-12-12T01:14:22.256 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-12-12T01:14:22.295 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-12-12T01:14:22.357 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-12-12T01:14:26.214 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:26.296 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T01:14:26.326 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off 2023-12-12T01:14:26.406 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2023-12-12T01:14:26.427 INFO:teuthology.orchestra.console.smithi100:Power off completed 2023-12-12T01:14:26.507 INFO:teuthology.orchestra.console.smithi137:Power off completed 2023-12-12T01:14:26.614 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} duration: 368.45347452163696 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: INTERNAL SERVER ERROR'', ''Error: Request to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 failed: 500 - HTTP Error 500: Internal Server Error'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=430b231412934625b724b7151e90f08f status: dead success: false 2023-12-12T01:14:26.614 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-12T01:14:26.700 INFO:teuthology.run:DEAD