2023-12-04T21:07:13.728 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:07:13.728 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368 --verbose --owner scheduled_yuriw@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368/orig.config.yaml 2023-12-04T21:07:13.760 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:07:13.840 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '7477368' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmNutBfleWESEsW3R1fl+yhe0Mo3soKHjAFd+wGjEsU9v0VnQPvxVJmUuu+J2LkfQ2StFvF6RtW4jDbtYOAwmw= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhQVw8fJkXHodFi8mYxeWZQsmlnLa2826g4B/PQr3nQqquRdwTQ+qqi053/Xrb/S01IclC46RliWbQgJ8wsqxQ= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:07:13.840 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:07:13.841 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:07:13.841 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:07:13.843 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:07:13.864 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:07:13.864 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:07:13.864 INFO:teuthology.packaging:ref: None 2023-12-04T21:07:13.864 INFO:teuthology.packaging:tag: None 2023-12-04T21:07:13.864 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:07:13.864 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:07:13.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:07:14.853 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:07:14.853 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:07:14.860 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:07:14.860 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:07:14.890 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:07:14.902 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:07:14.908 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:07:14.929 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:35.448454', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmNutBfleWESEsW3R1fl+yhe0Mo3soKHjAFd+wGjEsU9v0VnQPvxVJmUuu+J2LkfQ2StFvF6RtW4jDbtYOAwmw='} 2023-12-04T21:07:14.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:35.447297', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhQVw8fJkXHodFi8mYxeWZQsmlnLa2826g4B/PQr3nQqquRdwTQ+qqi053/Xrb/S01IclC46RliWbQgJ8wsqxQ='} 2023-12-04T21:07:14.952 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:07:14.960 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-12-04T21:07:14.960 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-12-04T21:07:14.961 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:07:15.032 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6982967ee0>, signals=[15]) 2023-12-04T21:07:15.033 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:07:15.039 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:07:15.039 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2023-12-04T21:07:15.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:07:15.110 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2023-12-04T21:07:15.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:07:15.178 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:07:15.184 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2023-12-04T21:07:15.206 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2023-12-04T21:07:15.206 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="8" 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="8" 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:07:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:07:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2023-12-04T21:07:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:07:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:07:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:07:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:07:15.267 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:07:15.268 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2023-12-04T21:07:15.290 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2023-12-04T21:07:15.313 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2023-12-04T21:07:15.313 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:07:15.374 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2023-12-04T21:07:15.375 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:07:15.375 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:07:15.375 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:07:15.375 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:07:15.375 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:07:15.375 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2023-12-04T21:07:15.398 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:07:15.409 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:07:15.432 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:07:15.432 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:07:15.434 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:07:15.454 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:07:15.462 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:07:15.462 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:07:15.494 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:07:15.525 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:07:15.578 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:07:15.626 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:07:15.627 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:07:15.627 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:07:15.627 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:07:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:07:15.647 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-12-04T21:07:15.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:07:15.648 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-12-04T21:07:15.706 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:07:15.707 INFO:teuthology.task.kernel:Running kernel on smithi082: 4.18.0-489.el8.x86_64 2023-12-04T21:07:15.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2023-12-04T21:07:15.709 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:07:15.709 INFO:teuthology.task.kernel:Running kernel on smithi111: 4.18.0-489.el8.x86_64 2023-12-04T21:07:15.709 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2023-12-04T21:07:54.825 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - AppStream 958 kB/s | 34 MB 00:36 2023-12-04T21:08:46.915 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - BaseOS 1.2 MB/s | 55 MB 00:45 2023-12-04T21:10:07.069 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 208 kB/s | 34 MB 02:48 2023-12-04T21:10:50.731 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 1.5 MB/s | 55 MB 00:37 2023-12-04T21:11:33.023 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras 115 B/s | 18 kB 02:38 2023-12-04T21:11:33.565 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-04T21:12:27.019 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:53 2023-12-04T21:12:27.020 INFO:teuthology.orchestra.run.smithi082.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:12:27.020 INFO:teuthology.orchestra.run.smithi082.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:12:27.020 INFO:teuthology.orchestra.run.smithi082.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:12:27.847 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-12-04T21:12:31.639 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 452 kB/s | 24 kB 00:00 2023-12-04T21:12:31.842 INFO:teuthology.orchestra.run.smithi082.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:12:35.127 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:12:35.128 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:12:35.419 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-12-04T21:12:35.420 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-12-04T21:12:35.420 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2023-12-04T21:12:35.420 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-12-04T21:12:35.420 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-12-04T21:12:35.420 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout:Install 3 Packages 2023-12-04T21:12:35.421 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T21:12:35.422 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 90 M 2023-12-04T21:12:35.422 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 96 M 2023-12-04T21:12:35.422 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-12-04T21:12:48.697 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:12:48.697 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:12:48.698 INFO:teuthology.orchestra.run.smithi082.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:12:51.378 INFO:teuthology.orchestra.run.smithi082.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 656 kB/s | 10 MB 00:15 2023-12-04T21:12:53.395 INFO:teuthology.orchestra.run.smithi082.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.0 MB/s | 36 MB 00:17 2023-12-04T21:12:54.445 INFO:teuthology.orchestra.run.smithi082.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.3 MB/s | 43 MB 00:18 2023-12-04T21:12:54.445 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:12:54.445 INFO:teuthology.orchestra.run.smithi082.stdout:Total 4.7 MB/s | 90 MB 00:19 2023-12-04T21:12:55.119 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-12-04T21:12:55.556 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-12-04T21:12:55.556 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-12-04T21:12:56.786 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-12-04T21:12:56.822 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-12-04T21:12:59.631 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-12-04T21:13:00.252 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:03.802 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:04.368 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:07.804 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:08.088 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:13:29.978 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:13:30.000 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:13:30.487 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:13:30.488 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:30.488 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T21:13:31.601 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-12-04T21:13:31.744 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:13:31.963 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:13:31.963 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:13:31.963 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:13:31.964 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:13:31.964 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:13:31.964 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-12-04T21:13:31.964 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:13:31.964 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:13:32.173 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:13:32.173 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:13:32.173 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.02 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 129.el8 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Group : System Environment/Base 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 9265586 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2023-12-04T21:13:32.208 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Relocations : (not relocatable) 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : CentOS BuildSystem 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:13:32.209 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:13:32.210 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2023-12-04T21:13:32.210 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-12-04T21:13:32.210 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:13:32.211 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:13:32.211 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:13:32.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:13:33.552 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2023-12-04T21:13:35.517 INFO:teuthology.orchestra.run.smithi082.stderr:done 2023-12-04T21:13:35.519 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2023-12-04T21:13:35.550 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.GxuLRKKmn3 2023-12-04T21:13:35.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GxuLRKKmn3 2023-12-04T21:13:35.624 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.GxuLRKKmn3 2023-12-04T21:13:35.769 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.GxuLRKKmn3 is 6KB 2023-12-04T21:13:35.818 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.GxuLRKKmn3 2023-12-04T21:13:35.836 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:13:35.907 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:13:35.907 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:13:35.907 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:13:35.908 INFO:teuthology.orchestra.run.smithi082.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:13:35.909 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:13:36.111 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2023-12-04T21:13:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 115 B/s | 18 kB 02:38 2023-12-04T21:13:38.959 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 5.8 kB/s | 6.9 kB 00:01 2023-12-04T21:14:06.114 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-12-04T21:14:06.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T21:14:06.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:24.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2023-12-04T21:14:33.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T21:14:33.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:38.495 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:55 2023-12-04T21:14:38.495 INFO:teuthology.orchestra.run.smithi111.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:14:38.495 INFO:teuthology.orchestra.run.smithi111.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:14:38.495 INFO:teuthology.orchestra.run.smithi111.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:15:38.495 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:15:50.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T21:15:50.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:55.887 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-12-04T21:15:56.214 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-12-04T21:15:56.214 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:15:56.214 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-12-04T21:15:56.233 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:15:56.234 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:15:56.234 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:15:56.234 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:15:57.234 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-12-04T21:15:57.235 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:15:57.297 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2023-12-04T21:15:57.312 DEBUG:teuthology.parallel:result is None 2023-12-04T21:17:16.583 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 101 kB/s | 16 MB 02:42 2023-12-04T21:17:20.398 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 263 kB/s | 24 kB 00:00 2023-12-04T21:17:20.603 INFO:teuthology.orchestra.run.smithi111.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:17:23.981 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:17:23.981 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:17:24.266 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-12-04T21:17:24.266 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-12-04T21:17:24.266 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2023-12-04T21:17:24.266 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-12-04T21:17:24.266 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout:Install 3 Packages 2023-12-04T21:17:24.267 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-04T21:17:24.268 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 90 M 2023-12-04T21:17:24.268 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 96 M 2023-12-04T21:17:24.268 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-12-04T21:19:52.529 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:19:52.529 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:19:52.529 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:19:55.829 INFO:teuthology.orchestra.run.smithi111.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 529 kB/s | 10 MB 00:19 2023-12-04T21:19:57.912 INFO:teuthology.orchestra.run.smithi111.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.7 MB/s | 36 MB 00:21 2023-12-04T21:19:58.721 INFO:teuthology.orchestra.run.smithi111.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 1.9 MB/s | 43 MB 00:22 2023-12-04T21:19:58.721 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:19:58.721 INFO:teuthology.orchestra.run.smithi111.stdout:Total 594 kB/s | 90 MB 02:34 2023-12-04T21:19:59.395 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-12-04T21:19:59.839 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-12-04T21:19:59.839 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-12-04T21:20:01.109 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-12-04T21:20:01.145 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-12-04T21:20:03.932 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-12-04T21:20:04.487 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:20:08.113 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:20:08.527 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:20:11.944 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:20:12.240 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:20:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:20:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:20:34.628 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:20:34.629 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:20:34.629 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-04T21:20:35.444 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-12-04T21:20:35.616 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:20:35.834 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:20:35.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:20:35.834 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:20:35.834 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:20:35.834 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:20:35.834 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-12-04T21:20:35.835 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:20:35.835 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:20:36.042 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:20:36.042 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:20:36.043 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.02 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 129.el8 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Group : System Environment/Base 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 9265586 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:20:36.076 INFO:teuthology.orchestra.run.smithi111.stdout:Relocations : (not relocatable) 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : CentOS BuildSystem 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-12-04T21:20:36.077 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:20:36.078 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:20:36.078 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:20:36.078 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:20:37.758 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2023-12-04T21:20:39.761 INFO:teuthology.orchestra.run.smithi111.stderr:done 2023-12-04T21:20:39.763 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-12-04T21:20:39.793 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.AinWFaWSZt 2023-12-04T21:20:39.793 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AinWFaWSZt 2023-12-04T21:20:39.866 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.AinWFaWSZt 2023-12-04T21:20:40.044 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.AinWFaWSZt is 6KB 2023-12-04T21:20:40.093 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.AinWFaWSZt 2023-12-04T21:20:40.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:20:40.187 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:20:40.188 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:20:40.188 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:20:40.188 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:20:40.189 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:20:40.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2023-12-04T21:21:10.353 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-12-04T21:21:10.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-04T21:21:10.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:21:38.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-12-04T21:21:47.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-04T21:21:47.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:21:56.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-12-04T21:22:08.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-04T21:22:08.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:22:41.018 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-12-04T21:22:41.439 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-12-04T21:22:41.439 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:22:41.439 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-12-04T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:22:41.496 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:22:41.497 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:22:41.497 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:22:42.497 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-12-04T21:22:42.498 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2023-12-04T21:22:42.573 DEBUG:teuthology.parallel:result is None 2023-12-04T21:22:42.573 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:22:42.580 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:22:42.580 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:22:42.583 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:22:42.603 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:22:42.617 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:22:42.650 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:22:42.650 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:22:42.653 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:22:42.707 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:22:42.714 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:22:42.714 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:22:42.716 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:22:42.749 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:22:42.753 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:22:42.760 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:22:42.764 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:22:42.765 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:22:42.771 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:22:42.772 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:22:42.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:22:42.834 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:22:42.842 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:22:42.842 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:22:42.873 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:22:42.892 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:22:46.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:22:46.404 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:22:46.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:22:46.496 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-12-04T21:22:46.496 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:22:46.565 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:22:46.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:22:46.667 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:22:46.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:22:46.800 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-12-04T21:22:46.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:22:46.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2023-12-04T21:22:46.873 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2023-12-04T21:22:46.953 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:22:46.972 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:22:47.277 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:22:47.285 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:22:47.285 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:22:47.321 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:22:47.355 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:22:47.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2023-12-04T21:22:47.469 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2023-12-04T21:22:47.471 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2023-12-04T21:22:47.550 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2023-12-04T21:22:47.551 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:22:47.552 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2023-12-04T21:22:47.576 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2023-12-04T21:22:47.577 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2023-12-04T21:22:47.646 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2023-12-04T21:22:47.646 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:22:47.646 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:22:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:22:47.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:22:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:22:47.704 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:22:47.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:22:47.745 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:22:47.773 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:22:47.782 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T21:22:47.787 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T21:22:52.918 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:22:52.934 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:22:52.935 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com,smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:38:51.718 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368/ansible_failures.yaml 2023-12-04T21:38:51.719 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:38:51.787 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=427d01d9642d45e5b5dbf27edb4f5590 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:38:51.788 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:38:51.796 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:38:51.796 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:38:51.844 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:38:51.933 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:38:52.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:38:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:52.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:38:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:52.088 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:38:52.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:38:52.140 INFO:teuthology.task.internal:Duration was 964.854682 seconds 2023-12-04T21:38:52.140 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:38:52.177 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:38:52.177 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:38:52.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:38:52.225 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:38:52.226 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:38:52.527 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:38:52.527 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2023-12-04T21:38:52.527 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:38:52.551 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2023-12-04T21:38:52.551 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:38:52.573 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:38:52.573 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:38:52.594 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:38:52.648 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:38:52.648 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:38:52.669 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:38:52.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:38:52.755 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:38:52.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:38:52.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:38:52.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:38:52.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:52.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:52.848 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2023-12-04T21:38:52.874 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2023-12-04T21:38:52.894 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:52.927 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:52.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:38:52.960 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:38:52.961 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368/remote/smithi082 2023-12-04T21:38:52.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:38:53.024 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368/remote/smithi111 2023-12-04T21:38:53.025 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:38:53.068 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:38:53.068 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:38:53.070 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:38:53.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:38:53.130 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:38:53.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:38:53.194 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:38:53.195 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:38:53.197 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:38:53.211 INFO:teuthology.orchestra.run.smithi082.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:38 /home/ubuntu/cephtest 2023-12-04T21:38:53.214 INFO:teuthology.orchestra.run.smithi111.stdout: 525362 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:38 /home/ubuntu/cephtest 2023-12-04T21:38:53.215 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:38:53.227 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:38:53.295 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:38:53.318 DEBUG:teuthology.nuke:shortname: smithi082 2023-12-04T21:38:53.319 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:38:53.345 DEBUG:teuthology.nuke:shortname: smithi111 2023-12-04T21:38:53.345 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:38:53.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:35.448454', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmNutBfleWESEsW3R1fl+yhe0Mo3soKHjAFd+wGjEsU9v0VnQPvxVJmUuu+J2LkfQ2StFvF6RtW4jDbtYOAwmw='} 2023-12-04T21:38:53.369 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:35.447297', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGhQVw8fJkXHodFi8mYxeWZQsmlnLa2826g4B/PQr3nQqquRdwTQ+qqi053/Xrb/S01IclC46RliWbQgJ8wsqxQ='} 2023-12-04T21:38:53.394 INFO:teuthology.orchestra.console.smithi082:Power off 2023-12-04T21:38:53.394 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:38:53.417 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2023-12-04T21:38:53.417 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:53.431 INFO:teuthology.orchestra.console.smithi111:Power off 2023-12-04T21:38:53.431 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:38:53.440 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-12-04T21:38:53.454 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-12-04T21:38:53.454 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:53.477 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-04T21:38:57.441 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:57.478 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:57.550 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2023-12-04T21:38:57.587 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-04T21:39:01.554 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:01.588 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:01.663 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2023-12-04T21:39:01.696 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-12-04T21:39:01.764 INFO:teuthology.orchestra.console.smithi082:Power off completed 2023-12-04T21:39:05.698 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:05.807 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-12-04T21:39:05.908 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-12-04T21:39:06.016 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 964.854681968689 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=427d01d9642d45e5b5dbf27edb4f5590 status: dead success: false 2023-12-04T21:39:06.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:39:06.083 INFO:teuthology.run:DEAD