2023-12-04T21:05:45.433 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:05:45.433 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi --verbose --description rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363 -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363/orig.config.yaml 2023-12-04T21:05:45.464 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:06:00.407 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} email: null first_in_suite: false job_id: '7477363' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBgNXtD3zehpODST2WfRLPVC9WbvHEf5+Mt/ikNxCMIeRvHG6f2ZzHYHcee8u5yoAlQBp+ZlPdZmaXO0J28/1k= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIT+KlCsWlenxyxaxT48fsBE7jzt1Y8NVUY3L0KriQ68oe5UN7fKV2J/k1zDXxy7QLjEYHAlJR7di/ZJf3cduWE= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:06:00.407 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:06:00.408 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:06:00.408 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:06:15.536 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:06:15.562 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:06:15.562 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:06:15.563 INFO:teuthology.packaging:ref: None 2023-12-04T21:06:15.563 INFO:teuthology.packaging:tag: None 2023-12-04T21:06:15.563 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:06:15.563 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:06:15.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:06:20.886 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:06:20.886 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:06:20.892 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:06:20.892 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:06:20.910 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:06:20.922 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:06:20.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:06:20.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:33.873241', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBgNXtD3zehpODST2WfRLPVC9WbvHEf5+Mt/ikNxCMIeRvHG6f2ZzHYHcee8u5yoAlQBp+ZlPdZmaXO0J28/1k='} 2023-12-04T21:06:20.971 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:33.874676', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIT+KlCsWlenxyxaxT48fsBE7jzt1Y8NVUY3L0KriQ68oe5UN7fKV2J/k1zDXxy7QLjEYHAlJR7di/ZJf3cduWE='} 2023-12-04T21:06:20.971 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:06:20.977 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-12-04T21:06:20.977 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-12-04T21:06:20.977 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:06:21.041 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5278c4bee0>, signals=[15]) 2023-12-04T21:06:21.041 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:06:21.046 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:06:21.046 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-12-04T21:06:21.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:06:21.119 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2023-12-04T21:06:21.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:06:21.190 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:06:21.196 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-12-04T21:06:21.212 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-12-04T21:06:21.213 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-12-04T21:06:21.268 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux" 2023-12-04T21:06:21.268 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.4 (Ootpa)" 2023-12-04T21:06:21.268 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.4" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-04T21:06:21.269 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-12-04T21:06:21.270 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-12-04T21:06:26.298 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2023-12-04T21:06:26.334 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2023-12-04T21:06:26.334 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Red Hat Enterprise Linux" 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8.4 (Ootpa)" 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:ID="rhel" 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="fedora" 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8.4" 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:06:26.390 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.redhat.com/" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout: 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-04T21:06:26.391 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-12-04T21:06:26.391 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2023-12-04T21:06:31.414 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:06:31.424 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:06:31.430 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:06:31.430 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:06:31.432 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:06:31.448 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:06:31.454 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:06:31.454 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:06:31.490 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:06:31.504 INFO:teuthology.orchestra.run.smithi031.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-12-04T21:06:31.506 INFO:teuthology.orchestra.run.smithi159.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-12-04T21:06:31.507 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:06:36.571 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:06:36.588 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:06:36.588 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:06:36.589 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:06:36.590 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:06:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:06:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:06:36.607 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2023-12-04T21:06:36.608 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-12-04T21:06:36.664 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-12-04T21:06:36.664 INFO:teuthology.task.kernel:Running kernel on smithi159: 4.18.0-305.25.1.el8_4.x86_64 2023-12-04T21:06:36.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2023-12-04T21:06:36.666 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-12-04T21:06:36.666 INFO:teuthology.task.kernel:Running kernel on smithi031: 4.18.0-305.25.1.el8_4.x86_64 2023-12-04T21:06:36.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-12-04T21:06:38.114 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-12-04T21:06:38.115 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-12-04T21:06:38.155 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2023-12-04T21:06:38.156 INFO:teuthology.orchestra.run.smithi159.stdout:Unable to read consumer identity 2023-12-04T21:07:31.494 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:52 2023-12-04T21:07:31.495 INFO:teuthology.orchestra.run.smithi159.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:07:31.495 INFO:teuthology.orchestra.run.smithi159.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:07:31.495 INFO:teuthology.orchestra.run.smithi159.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:07:31.521 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:52 2023-12-04T21:07:31.521 INFO:teuthology.orchestra.run.smithi031.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:07:31.521 INFO:teuthology.orchestra.run.smithi031.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:07:31.521 INFO:teuthology.orchestra.run.smithi031.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:11:52.897 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 62 kB/s | 16 MB 04:21 2023-12-04T21:11:53.113 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 62 kB/s | 16 MB 04:21 2023-12-04T21:11:56.182 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-12-04T21:11:56.261 INFO:teuthology.orchestra.run.smithi159.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:11:56.601 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-12-04T21:11:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:11:56.897 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-12-04T21:11:56.897 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-12-04T21:11:56.897 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-12-04T21:11:57.048 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2023-12-04T21:11:57.049 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2023-12-04T21:11:57.050 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2023-12-04T21:11:57.165 DEBUG:teuthology.orchestra.run.smithi159:> echo no | sudo yum reinstall kernel || true 2023-12-04T21:11:57.266 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-12-04T21:11:57.266 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-12-04T21:11:57.267 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-12-04T21:11:57.423 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-12-04T21:11:57.424 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-12-04T21:11:57.424 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-12-04T21:11:57.519 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2023-12-04T21:11:57.519 INFO:teuthology.orchestra.run.smithi159.stdout:Unable to read consumer identity 2023-12-04T21:11:57.555 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2023-12-04T21:11:57.931 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-12-04T21:11:57.931 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-12-04T21:12:53.030 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:55 2023-12-04T21:12:53.031 INFO:teuthology.orchestra.run.smithi159.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:12:53.031 INFO:teuthology.orchestra.run.smithi159.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:12:53.031 INFO:teuthology.orchestra.run.smithi159.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:12:53.056 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:54 2023-12-04T21:12:53.056 INFO:teuthology.orchestra.run.smithi031.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:12:53.057 INFO:teuthology.orchestra.run.smithi031.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:12:53.057 INFO:teuthology.orchestra.run.smithi031.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:12:53.150 INFO:teuthology.orchestra.run.smithi159.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:12:53.150 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Dec 2023 09:11:56 PM UTC. 2023-12-04T21:12:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:12:53.178 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Dec 2023 09:11:56 PM UTC. 2023-12-04T21:12:53.321 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-04T21:12:53.322 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:12:53.322 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:12:53.335 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-04T21:12:53.335 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:12:53.335 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:12:53.347 INFO:teuthology.orchestra.run.smithi159.stderr:Error: No packages marked for reinstall. 2023-12-04T21:12:53.366 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-12-04T21:12:53.396 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum reinstall -y kernel || true 2023-12-04T21:12:53.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2023-12-04T21:12:53.741 INFO:teuthology.orchestra.run.smithi159.stdout:Updating Subscription Management repositories. 2023-12-04T21:12:53.741 INFO:teuthology.orchestra.run.smithi159.stdout:Unable to read consumer identity 2023-12-04T21:12:53.785 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-12-04T21:12:53.785 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-12-04T21:13:43.489 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:49 2023-12-04T21:13:43.489 INFO:teuthology.orchestra.run.smithi031.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:13:43.490 INFO:teuthology.orchestra.run.smithi031.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:13:43.490 INFO:teuthology.orchestra.run.smithi031.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:13:43.526 INFO:teuthology.orchestra.run.smithi159.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:49 2023-12-04T21:13:43.527 INFO:teuthology.orchestra.run.smithi159.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:13:43.527 INFO:teuthology.orchestra.run.smithi159.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:13:43.527 INFO:teuthology.orchestra.run.smithi159.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:13:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:13:43.609 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:47 ago on Mon 04 Dec 2023 09:11:56 PM UTC. 2023-12-04T21:13:43.652 INFO:teuthology.orchestra.run.smithi159.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:13:43.652 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:47 ago on Mon 04 Dec 2023 09:11:56 PM UTC. 2023-12-04T21:13:43.744 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-04T21:13:43.744 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:13:43.744 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:13:43.756 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-12-04T21:13:43.812 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:13:43.818 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-04T21:13:43.818 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:13:43.819 INFO:teuthology.orchestra.run.smithi159.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-04T21:13:43.833 INFO:teuthology.orchestra.run.smithi159.stderr:Error: No packages marked for reinstall. 2023-12-04T21:13:43.883 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:13:43.943 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:43.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:43.943 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:43.943 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:13:43.943 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:13:43.943 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-12-04T21:13:43.944 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:13:43.944 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:13:44.009 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.009 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.010 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.010 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:13:44.010 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:13:44.010 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-12-04T21:13:44.010 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:13:44.010 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:13:44.074 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.074 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-12-04T21:13:44.102 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.02 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 99.el8_4.1 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Group : System Environment/Base 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 9264631 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Relocations : (not relocatable) 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : Red Hat, Inc. 2023-12-04T21:13:44.103 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : Red Hat, Inc. 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-12-04T21:13:44.104 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:13:44.105 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:13:44.105 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:13:44.135 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.136 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2023-12-04T21:13:44.163 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2023-12-04T21:13:44.163 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2023-12-04T21:13:44.163 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.02 2023-12-04T21:13:44.163 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 99.el8_4.1 2023-12-04T21:13:44.163 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2023-12-04T21:13:44.163 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:Group : System Environment/Base 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 9264631 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-12-04T21:13:44.164 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout:Relocations : (not relocatable) 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : Red Hat, Inc. 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : Red Hat, Inc. 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2023-12-04T21:13:44.165 INFO:teuthology.orchestra.run.smithi159.stdout: 2023-12-04T21:13:44.166 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:13:44.166 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:13:44.166 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:13:44.166 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2023-12-04T21:13:44.166 INFO:teuthology.orchestra.run.smithi159.stdout: 2023-12-04T21:13:44.166 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:13:44.167 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:13:44.168 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:44.168 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:13:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-12-04T21:13:45.035 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2023-12-04T21:13:46.866 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-12-04T21:13:46.868 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-12-04T21:13:46.886 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.nrkkTzwnbd 2023-12-04T21:13:46.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nrkkTzwnbd 2023-12-04T21:13:46.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.nrkkTzwnbd 2023-12-04T21:13:47.162 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.nrkkTzwnbd is 6KB 2023-12-04T21:13:47.211 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.nrkkTzwnbd 2023-12-04T21:13:47.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:13:47.330 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-12-04T21:13:47.330 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-12-04T21:13:47.330 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-12-04T21:13:47.330 INFO:teuthology.orchestra.run.smithi031.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-12-04T21:13:47.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:47.359 INFO:teuthology.orchestra.run.smithi159.stderr:done 2023-12-04T21:13:47.361 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2023-12-04T21:13:47.375 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.2rhfhTzimr 2023-12-04T21:13:47.375 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2rhfhTzimr 2023-12-04T21:13:47.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-12-04T21:13:47.445 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.2rhfhTzimr 2023-12-04T21:13:47.607 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.2rhfhTzimr is 6KB 2023-12-04T21:13:47.655 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.2rhfhTzimr 2023-12-04T21:13:47.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:13:47.754 INFO:teuthology.orchestra.run.smithi159.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-12-04T21:13:47.754 INFO:teuthology.orchestra.run.smithi159.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-12-04T21:13:47.754 INFO:teuthology.orchestra.run.smithi159.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-12-04T21:13:47.754 INFO:teuthology.orchestra.run.smithi159.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-12-04T21:13:47.755 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-12-04T21:13:47.897 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2023-12-04T21:14:17.470 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-12-04T21:14:17.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-12-04T21:14:17.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:17.901 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-12-04T21:14:17.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2023-12-04T21:14:17.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:36.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-12-04T21:14:36.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-12-04T21:14:45.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-12-04T21:14:45.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:50.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2023-12-04T21:14:50.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:58.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2023-12-04T21:15:08.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-12-04T21:15:10.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2023-12-04T21:15:10.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:20.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-12-04T21:15:20.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:21.007 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-12-04T21:15:21.323 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-12-04T21:15:21.323 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-12-04T21:15:21.324 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-12-04T21:15:21.338 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-425.3.1.el8.x86_64 2023-12-04T21:15:21.340 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:15:21.340 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:15:21.341 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-12-04T21:15:22.342 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-12-04T21:15:22.342 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:15:22.421 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-12-04T21:15:22.435 DEBUG:teuthology.parallel:result is None 2023-12-04T21:15:42.038 DEBUG:teuthology.orchestra.run.smithi159:> true 2023-12-04T21:15:42.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2023-12-04T21:15:42.305 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-12-04T21:15:42.305 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2023-12-04T21:15:42.363 INFO:teuthology.orchestra.run.smithi159.stdout:4.18.0-425.3.1.el8.x86_64 2023-12-04T21:15:42.363 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-12-04T21:15:42.363 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:15:42.363 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-12-04T21:15:43.364 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-12-04T21:15:43.365 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:15:43.476 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2023-12-04T21:15:43.494 DEBUG:teuthology.parallel:result is None 2023-12-04T21:15:43.494 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:15:43.501 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:15:43.501 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:15:43.505 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:15:43.521 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:15:43.530 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:15:43.541 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:15:43.541 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:15:43.564 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:15:43.622 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:15:43.630 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:15:43.630 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:15:43.633 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:15:43.664 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:15:43.673 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:15:43.675 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:15:43.685 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:15:43.686 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:15:43.696 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:15:43.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:15:43.729 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:15:43.754 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:15:43.767 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:15:43.768 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:15:43.796 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:15:43.812 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:15:43.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:15:43.936 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:15:44.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:15:44.060 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-12-04T21:15:44.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:15:44.125 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:15:44.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:15:44.221 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:15:44.308 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:15:44.351 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2023-12-04T21:15:44.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:15:44.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-12-04T21:15:44.423 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2023-12-04T21:15:44.486 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:15:44.517 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:15:44.891 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:15:44.902 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:15:44.902 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:15:44.924 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:15:44.938 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:15:44.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-12-04T21:15:45.004 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: [ OK ] 2023-12-04T21:15:45.006 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2023-12-04T21:15:45.404 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: [ OK ] 2023-12-04T21:15:45.406 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:15:45.407 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-12-04T21:15:45.430 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-12-04T21:15:45.430 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2023-12-04T21:15:45.469 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2023-12-04T21:15:45.469 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:15:45.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:15:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:15:45.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:15:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:15:45.657 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:15:45.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:15:45.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:15:45.845 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:15:45.859 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T21:15:45.864 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T21:15:46.046 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:15:46.060 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:15:46.061 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:38:19.428 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363/ansible_failures.yaml 2023-12-04T21:38:19.430 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:38:19.497 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e27d82072c742c38c30316b8337ac71 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:38:19.498 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:38:19.506 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:38:19.506 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:38:19.521 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:38:24.634 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:38:24.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:38:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:24.750 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:38:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:24.780 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:38:24.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:38:24.812 INFO:teuthology.task.internal:Duration was 1359.909792 seconds 2023-12-04T21:38:24.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:38:24.827 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:38:24.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:38:24.830 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:38:24.874 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:38:24.875 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:38:25.121 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:38:25.122 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-12-04T21:38:25.122 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:38:25.142 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2023-12-04T21:38:25.143 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:38:25.161 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:38:25.161 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:38:25.185 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:38:25.236 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:38:25.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:38:25.262 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:38:25.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:38:25.347 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:38:25.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:38:25.372 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:38:25.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:38:25.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:25.414 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:25.436 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-12-04T21:38:25.460 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2023-12-04T21:38:25.473 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:25.504 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:25.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:25.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:38:25.536 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:38:25.538 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363/remote/smithi031 2023-12-04T21:38:25.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:38:25.614 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363/remote/smithi159 2023-12-04T21:38:25.615 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:38:25.662 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:38:25.662 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:38:25.664 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:38:25.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:38:25.719 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:38:25.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:38:25.730 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:38:25.730 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:38:25.732 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:38:25.744 INFO:teuthology.orchestra.run.smithi031.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:38 /home/ubuntu/cephtest 2023-12-04T21:38:25.763 INFO:teuthology.orchestra.run.smithi159.stdout: 262790 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:38 /home/ubuntu/cephtest 2023-12-04T21:38:25.764 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:38:25.774 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:38:25.796 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:38:25.823 DEBUG:teuthology.nuke:shortname: smithi031 2023-12-04T21:38:25.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:38:25.846 DEBUG:teuthology.nuke:shortname: smithi159 2023-12-04T21:38:25.846 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:38:25.850 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:33.873241', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBgNXtD3zehpODST2WfRLPVC9WbvHEf5+Mt/ikNxCMIeRvHG6f2ZzHYHcee8u5yoAlQBp+ZlPdZmaXO0J28/1k='} 2023-12-04T21:38:25.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477363', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:33.874676', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIT+KlCsWlenxyxaxT48fsBE7jzt1Y8NVUY3L0KriQ68oe5UN7fKV2J/k1zDXxy7QLjEYHAlJR7di/ZJf3cduWE='} 2023-12-04T21:38:25.900 INFO:teuthology.orchestra.console.smithi031:Power off 2023-12-04T21:38:25.900 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:38:25.924 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-12-04T21:38:25.924 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:25.940 INFO:teuthology.orchestra.console.smithi159:Power off 2023-12-04T21:38:25.940 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:38:25.949 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-12-04T21:38:25.962 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2023-12-04T21:38:25.962 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:25.986 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-12-04T21:38:29.951 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:29.987 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:30.063 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-12-04T21:38:30.096 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2023-12-04T21:38:34.068 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:34.097 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:34.181 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-12-04T21:38:34.207 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2023-12-04T21:38:34.283 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-12-04T21:38:34.308 INFO:teuthology.orchestra.console.smithi159:Power off completed 2023-12-04T21:38:34.414 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} duration: 1359.9097921848297 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Connection timed out'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8e27d82072c742c38c30316b8337ac71 status: dead success: false 2023-12-04T21:38:34.415 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:38:34.506 INFO:teuthology.run:DEAD