2023-12-04T21:06:16.224 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:06:16.224 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357 --verbose --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --description rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357/orig.config.yaml 2023-12-04T21:06:16.267 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:06:20.710 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '7477357' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+AO/i9WbvmYhAZegg5eOJC1rbPxSkeN6PPQF6tGG02/EV1vPiV1Tx7hydGjoQwsVYQpoziL08aTMkoO0TjKB4= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOtemLFzQG1ig3w2Lg+dgC1DvQ4rTEkgEqP9A+DrsxbOaMCURuWMtTCwk21c6t22heR6KvRCQKksaltZKxrqIw= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:06:20.711 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:06:20.711 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:06:20.711 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:06:20.713 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:06:20.736 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:06:20.736 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:06:20.736 INFO:teuthology.packaging:ref: None 2023-12-04T21:06:20.737 INFO:teuthology.packaging:tag: None 2023-12-04T21:06:20.737 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:06:20.737 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:06:20.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:06:25.806 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:06:25.806 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:06:25.816 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:06:25.816 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:06:25.849 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:06:25.862 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:06:25.882 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:06:30.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:03.465664', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+AO/i9WbvmYhAZegg5eOJC1rbPxSkeN6PPQF6tGG02/EV1vPiV1Tx7hydGjoQwsVYQpoziL08aTMkoO0TjKB4='} 2023-12-04T21:06:36.573 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:03.467337', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOtemLFzQG1ig3w2Lg+dgC1DvQ4rTEkgEqP9A+DrsxbOaMCURuWMtTCwk21c6t22heR6KvRCQKksaltZKxrqIw='} 2023-12-04T21:06:36.573 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:06:36.579 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-12-04T21:06:36.579 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-12-04T21:06:36.579 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:06:36.676 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0514ed4ee0>, signals=[15]) 2023-12-04T21:06:36.677 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:06:36.683 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:06:36.683 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2023-12-04T21:06:36.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:06:36.752 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2023-12-04T21:06:36.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:06:36.822 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:06:36.828 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2023-12-04T21:06:36.848 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2023-12-04T21:06:36.848 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="8" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="8" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:06:36.907 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:06:36.908 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:06:36.908 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2023-12-04T21:06:36.930 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2023-12-04T21:06:36.949 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2023-12-04T21:06:36.950 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2023-12-04T21:06:37.008 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2023-12-04T21:06:37.008 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8" 2023-12-04T21:06:37.008 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2023-12-04T21:06:37.008 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:06:37.008 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:06:37.009 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:06:37.009 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2023-12-04T21:06:37.029 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:06:37.038 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:06:37.096 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:06:37.096 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:06:37.098 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:06:37.117 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:06:37.129 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:06:37.130 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:06:37.157 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:06:37.186 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:06:42.239 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:06:42.254 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:06:42.254 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:06:42.254 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:06:42.255 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:06:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:06:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:06:42.277 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-12-04T21:06:42.277 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2023-12-04T21:06:42.338 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:06:42.339 INFO:teuthology.task.kernel:Running kernel on smithi193: 4.18.0-489.el8.x86_64 2023-12-04T21:06:42.339 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2023-12-04T21:06:42.340 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:06:42.341 INFO:teuthology.task.kernel:Running kernel on smithi094: 4.18.0-489.el8.x86_64 2023-12-04T21:06:42.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2023-12-04T21:09:32.251 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - AppStream 209 kB/s | 34 MB 02:47 2023-12-04T21:09:32.308 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - AppStream 210 kB/s | 34 MB 02:47 2023-12-04T21:12:24.450 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - BaseOS 339 kB/s | 55 MB 02:45 2023-12-04T21:12:25.200 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - BaseOS 337 kB/s | 55 MB 02:46 2023-12-04T21:15:10.734 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - Extras 116 B/s | 18 kB 02:37 2023-12-04T21:15:10.866 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - Extras 115 B/s | 18 kB 02:38 2023-12-04T21:15:11.417 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-04T21:16:05.881 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:54 2023-12-04T21:16:05.881 INFO:teuthology.orchestra.run.smithi094.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:16:05.882 INFO:teuthology.orchestra.run.smithi094.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:16:05.882 INFO:teuthology.orchestra.run.smithi094.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:17:23.514 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - Extras common packages 54 B/s | 6.9 kB 02:12 2023-12-04T21:18:19.380 INFO:teuthology.orchestra.run.smithi193.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:55 2023-12-04T21:18:19.381 INFO:teuthology.orchestra.run.smithi193.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:18:19.381 INFO:teuthology.orchestra.run.smithi193.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:18:19.381 INFO:teuthology.orchestra.run.smithi193.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:18:20.193 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 122 kB/s | 16 MB 02:14 2023-12-04T21:18:23.974 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 424 kB/s | 24 kB 00:00 2023-12-04T21:18:24.176 INFO:teuthology.orchestra.run.smithi094.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:18:25.751 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 2.5 MB/s | 16 MB 00:06 2023-12-04T21:18:27.531 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:18:27.531 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:18:27.818 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-12-04T21:18:27.819 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2023-12-04T21:18:27.820 INFO:teuthology.orchestra.run.smithi094.stdout:Install 3 Packages 2023-12-04T21:18:27.820 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-12-04T21:18:27.820 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 90 M 2023-12-04T21:18:27.820 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 96 M 2023-12-04T21:18:27.820 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-12-04T21:18:29.539 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 626 kB/s | 24 kB 00:00 2023-12-04T21:18:29.744 INFO:teuthology.orchestra.run.smithi193.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:18:33.022 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:18:33.023 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:18:33.303 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout:Install 3 Packages 2023-12-04T21:18:33.304 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-12-04T21:18:33.305 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 90 M 2023-12-04T21:18:33.305 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 96 M 2023-12-04T21:18:33.305 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2023-12-04T21:18:41.297 INFO:teuthology.orchestra.run.smithi094.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:18:41.297 INFO:teuthology.orchestra.run.smithi094.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:18:41.297 INFO:teuthology.orchestra.run.smithi094.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:18:43.965 INFO:teuthology.orchestra.run.smithi094.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 648 kB/s | 10 MB 00:15 2023-12-04T21:18:45.815 INFO:teuthology.orchestra.run.smithi094.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.0 MB/s | 36 MB 00:17 2023-12-04T21:18:49.841 INFO:teuthology.orchestra.run.smithi094.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.0 MB/s | 43 MB 00:21 2023-12-04T21:18:49.842 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:18:49.842 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.1 MB/s | 90 MB 00:22 2023-12-04T21:18:50.528 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-12-04T21:18:50.992 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-12-04T21:18:50.992 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-12-04T21:18:52.246 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-12-04T21:18:52.281 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-12-04T21:18:55.086 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-12-04T21:18:55.565 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:59.195 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:59.564 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:19:02.997 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:19:03.310 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:25.230 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:25.254 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:19:25.593 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-12-04T21:19:26.635 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-12-04T21:19:26.828 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:19:27.083 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:19:27.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:19:27.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:19:27.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:19:27.084 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:19:27.084 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-12-04T21:19:27.084 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:19:27.085 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:19:27.293 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:19:27.294 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:19:27.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.02 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 129.el8 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Group : System Environment/Base 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 9265586 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2023-12-04T21:19:27.328 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Relocations : (not relocatable) 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : CentOS BuildSystem 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:19:27.329 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2023-12-04T21:19:27.330 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-12-04T21:19:27.330 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:19:27.331 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:19:27.349 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:19:27.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:19:28.849 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2023-12-04T21:19:30.811 INFO:teuthology.orchestra.run.smithi094.stderr:done 2023-12-04T21:19:30.813 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2023-12-04T21:19:30.851 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.5YtmqUO730 2023-12-04T21:19:30.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5YtmqUO730 2023-12-04T21:19:30.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.5YtmqUO730 2023-12-04T21:19:31.080 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.5YtmqUO730 is 6KB 2023-12-04T21:19:31.128 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.5YtmqUO730 2023-12-04T21:19:31.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:19:31.221 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:19:31.221 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:19:31.221 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:19:31.221 INFO:teuthology.orchestra.run.smithi094.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:19:31.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:19:31.373 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2023-12-04T21:20:01.376 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-12-04T21:20:01.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-12-04T21:20:01.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:20:24.930 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2023-12-04T21:20:33.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-12-04T21:20:33.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:20:57.429 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:20:57.430 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:20:57.430 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:21:00.270 INFO:teuthology.orchestra.run.smithi193.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 649 kB/s | 10 MB 00:15 2023-12-04T21:21:02.653 INFO:teuthology.orchestra.run.smithi193.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.0 MB/s | 36 MB 00:18 2023-12-04T21:21:08.910 INFO:teuthology.orchestra.run.smithi193.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 1.8 MB/s | 43 MB 00:24 2023-12-04T21:21:08.911 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:21:08.911 INFO:teuthology.orchestra.run.smithi193.stdout:Total 589 kB/s | 90 MB 02:35 2023-12-04T21:21:09.581 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2023-12-04T21:21:10.007 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2023-12-04T21:21:10.007 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2023-12-04T21:21:11.252 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2023-12-04T21:21:11.285 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2023-12-04T21:21:14.069 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2023-12-04T21:21:14.571 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:21:18.109 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:21:18.462 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:21:21.888 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:21:22.184 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:21:33.936 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:21:43.955 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:21:43.978 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:21:44.323 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:21:44.323 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:21:44.323 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-12-04T21:21:45.118 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2023-12-04T21:21:45.252 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:21:45.465 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:21:45.466 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:21:45.466 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:21:45.466 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:21:45.466 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:21:45.466 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-12-04T21:21:45.466 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:21:45.466 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:21:45.675 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:21:45.675 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:21:45.675 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2023-12-04T21:21:45.709 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2023-12-04T21:21:45.709 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2023-12-04T21:21:45.709 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.02 2023-12-04T21:21:45.709 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 129.el8 2023-12-04T21:21:45.709 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2023-12-04T21:21:45.709 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Group : System Environment/Base 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 9265586 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Relocations : (not relocatable) 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : CentOS BuildSystem 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2023-12-04T21:21:45.710 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-12-04T21:21:45.711 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:21:45.712 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:21:45.712 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:21:45.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:21:45.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-12-04T21:21:45.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:21:50.945 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2023-12-04T21:21:50.945 INFO:teuthology.orchestra.run.smithi193.stderr:done 2023-12-04T21:21:50.945 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-12-04T21:21:50.973 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.tdtrv0IgSQ 2023-12-04T21:21:50.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tdtrv0IgSQ 2023-12-04T21:21:51.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.tdtrv0IgSQ 2023-12-04T21:21:51.174 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.tdtrv0IgSQ is 6KB 2023-12-04T21:21:51.223 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.tdtrv0IgSQ 2023-12-04T21:21:51.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:21:51.273 DEBUG:teuthology.orchestra.run.smithi094:> true 2023-12-04T21:21:51.322 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:21:51.322 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:21:51.322 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:21:51.322 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:21:51.323 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:21:51.491 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2023-12-04T21:21:51.550 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-12-04T21:21:51.551 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:21:51.551 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2023-12-04T21:21:51.570 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:21:51.571 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:21:51.571 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:21:51.571 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:21:52.572 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-12-04T21:21:52.572 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:21:52.653 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2023-12-04T21:21:52.666 DEBUG:teuthology.parallel:result is None 2023-12-04T21:22:21.494 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-12-04T21:22:21.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-12-04T21:22:21.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:22:44.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-12-04T21:22:53.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-12-04T21:22:53.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:23:53.967 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:24:05.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-12-04T21:24:05.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:24:11.232 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-12-04T21:24:11.540 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-12-04T21:24:11.540 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:24:11.541 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-12-04T21:24:11.560 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:24:11.560 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:24:11.560 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:24:11.560 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:24:12.561 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-12-04T21:24:12.561 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:24:12.653 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2023-12-04T21:24:12.665 DEBUG:teuthology.parallel:result is None 2023-12-04T21:24:12.666 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:24:12.673 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:24:12.673 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:24:12.676 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:24:12.696 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:24:12.704 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:24:12.741 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:24:12.741 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:24:12.743 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:24:12.921 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:24:12.928 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:24:12.929 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:24:12.931 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:24:12.971 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:24:12.974 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:24:12.982 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:24:12.986 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:24:12.987 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:24:12.993 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:24:12.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:24:13.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:24:13.057 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:24:13.066 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:24:13.066 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:24:13.097 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:24:13.117 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:24:13.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:24:13.405 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:24:13.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:24:13.494 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-12-04T21:24:13.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:24:13.561 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:24:13.610 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:24:13.670 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:24:13.716 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:24:13.763 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-12-04T21:24:13.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:24:13.834 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2023-12-04T21:24:13.836 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2023-12-04T21:24:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:24:13.953 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:24:14.287 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:24:14.295 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:24:14.295 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:24:14.321 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:24:14.354 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:24:14.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2023-12-04T21:24:14.431 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2023-12-04T21:24:14.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2023-12-04T21:24:14.525 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2023-12-04T21:24:14.527 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:24:14.528 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2023-12-04T21:24:14.561 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2023-12-04T21:24:14.561 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2023-12-04T21:24:14.622 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2023-12-04T21:24:14.622 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:24:14.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:24:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:24:14.652 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:24:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:24:14.679 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:24:14.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:24:14.720 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:24:14.746 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:24:14.755 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T21:24:14.760 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T21:24:19.883 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:24:19.899 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:24:19.900 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:42:13.167 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357/ansible_failures.yaml 2023-12-04T21:42:13.169 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:42:13.241 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=efa7a9dcb00d4c5dab8809ada6dbf303 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:42:13.243 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:42:13.251 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:42:13.252 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:42:13.279 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:42:18.389 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:42:18.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:42:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:18.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:42:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:18.544 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:42:18.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:42:18.576 INFO:teuthology.task.internal:Duration was 1084.280293 seconds 2023-12-04T21:42:18.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:42:18.613 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:42:18.613 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:42:18.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:42:18.662 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:42:18.662 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:42:19.039 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:42:19.039 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2023-12-04T21:42:19.039 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:42:19.066 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2023-12-04T21:42:19.066 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:42:19.091 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:42:19.091 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:42:19.109 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:42:19.171 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:42:19.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:42:19.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:42:19.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:42:19.283 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:42:19.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:42:19.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:42:19.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:42:19.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:19.356 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:19.380 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2023-12-04T21:42:19.407 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2023-12-04T21:42:19.448 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:19.473 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:19.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:42:19.516 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:42:19.517 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357/remote/smithi094 2023-12-04T21:42:19.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:42:19.570 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357/remote/smithi193 2023-12-04T21:42:19.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:42:19.637 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:42:19.638 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:42:19.641 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:42:19.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:42:19.669 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:42:19.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:42:19.710 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:42:19.710 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:42:19.713 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:42:19.728 INFO:teuthology.orchestra.run.smithi094.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:42 /home/ubuntu/cephtest 2023-12-04T21:42:19.730 INFO:teuthology.orchestra.run.smithi193.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:42 /home/ubuntu/cephtest 2023-12-04T21:42:19.732 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:42:19.745 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:42:19.811 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:42:24.846 DEBUG:teuthology.nuke:shortname: smithi094 2023-12-04T21:42:24.848 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:42:26.361 DEBUG:teuthology.nuke:shortname: smithi193 2023-12-04T21:42:26.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:42:26.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:03.465664', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+AO/i9WbvmYhAZegg5eOJC1rbPxSkeN6PPQF6tGG02/EV1vPiV1Tx7hydGjoQwsVYQpoziL08aTMkoO0TjKB4='} 2023-12-04T21:42:26.386 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:03.467337', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOtemLFzQG1ig3w2Lg+dgC1DvQ4rTEkgEqP9A+DrsxbOaMCURuWMtTCwk21c6t22heR6KvRCQKksaltZKxrqIw='} 2023-12-04T21:42:26.416 INFO:teuthology.orchestra.console.smithi094:Power off 2023-12-04T21:42:26.416 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:42:26.437 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2023-12-04T21:42:26.437 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:26.452 INFO:teuthology.orchestra.console.smithi193:Power off 2023-12-04T21:42:26.452 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:42:26.459 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2023-12-04T21:42:26.472 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2023-12-04T21:42:26.472 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:26.495 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2023-12-04T21:42:30.461 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:30.496 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:31.570 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:42:31.602 DEBUG:teuthology.orchestra.console.smithi193:check power output: 2023-12-04T21:42:35.571 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:35.603 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:36.682 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:42:36.713 DEBUG:teuthology.orchestra.console.smithi193:check power output: 2023-12-04T21:42:40.683 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:40.714 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:41.792 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:42:41.822 DEBUG:teuthology.orchestra.console.smithi193:check power output: 2023-12-04T21:42:45.793 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:45.823 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:46.905 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:42:46.932 DEBUG:teuthology.orchestra.console.smithi193:check power output: 2023-12-04T21:42:50.910 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:50.933 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:52.023 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:42:52.042 DEBUG:teuthology.orchestra.console.smithi193:check power output: 2023-12-04T21:42:56.024 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:56.043 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:42:56.153 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2023-12-04T21:42:56.253 INFO:teuthology.orchestra.console.smithi193:Power off completed 2023-12-04T21:42:57.134 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:43:01.136 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:02.246 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:43:06.249 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:07.359 DEBUG:teuthology.orchestra.console.smithi094:check power output: 2023-12-04T21:43:11.360 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:11.472 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2023-12-04T21:43:11.572 INFO:teuthology.orchestra.console.smithi094:Power off completed 2023-12-04T21:43:11.679 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} duration: 1084.2802925109863 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=efa7a9dcb00d4c5dab8809ada6dbf303 status: dead success: false 2023-12-04T21:43:11.679 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:43:11.761 INFO:teuthology.run:DEAD