2023-12-04T21:05:03.144 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:05:03.144 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} --verbose --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365 -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365/orig.config.yaml 2023-12-04T21:05:03.179 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:05:07.739 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} email: null first_in_suite: false job_id: '7477365' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 rgw max chunk size: 4194304 rgw obj stripe size: 6291456 setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 lockdep: true osd: debug ms: 1 debug osd: 20 lockdep: true osd objectstore: filestore osd shutdown pgref assert: true osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) - \(OSD_SLOW_PING_TIME sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 rgw: compression type: random ec-data-pool: false frontend: beast storage classes: LUKEWARM, FROZEN selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMMwROLMYke0SBoH/oM1wcptn1WJqmMmR1hYKvOCw8NIK2fd2r9BPA+6HV+C2pH1mfoRNw2nqchMtSw/AN+ZsCk= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKL2lk7VvfQkRyVlMGdlm7qCGlg/26UOP/TwnUauVAyRrhQN+EdsBvFd9L+SJWgo8TPePobSuZtltKTbuolFqtg= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_rgw_throttle.sh - rgw/test_librgw_file.sh - ragweed: client.0: default-branch: ceph-pacific rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-pacific force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: ceph-pacific rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:05:07.739 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:05:07.740 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:05:07.740 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:05:07.742 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:05:07.767 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:05:07.767 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:05:07.767 INFO:teuthology.packaging:ref: None 2023-12-04T21:05:07.767 INFO:teuthology.packaging:tag: None 2023-12-04T21:05:07.767 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:05:07.767 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:05:07.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:05:18.239 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:05:18.239 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:05:18.247 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:05:18.247 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:05:18.255 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:05:18.269 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:05:18.276 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:05:28.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:34.516514', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMMwROLMYke0SBoH/oM1wcptn1WJqmMmR1hYKvOCw8NIK2fd2r9BPA+6HV+C2pH1mfoRNw2nqchMtSw/AN+ZsCk='} 2023-12-04T21:05:33.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:34.518582', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKL2lk7VvfQkRyVlMGdlm7qCGlg/26UOP/TwnUauVAyRrhQN+EdsBvFd9L+SJWgo8TPePobSuZtltKTbuolFqtg='} 2023-12-04T21:05:33.340 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:05:33.347 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-04T21:05:33.348 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-04T21:05:33.348 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:05:33.423 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f281505fee0>, signals=[15]) 2023-12-04T21:05:33.423 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:05:33.431 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:05:33.431 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-12-04T21:05:33.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:05:33.508 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2023-12-04T21:05:33.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:05:33.576 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:05:33.582 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-12-04T21:05:33.602 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-12-04T21:05:33.603 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-12-04T21:05:33.662 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2023-12-04T21:05:33.662 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8" 2023-12-04T21:05:33.662 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2023-12-04T21:05:33.662 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:05:33.662 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:05:33.663 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:05:33.664 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-12-04T21:05:33.689 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2023-12-04T21:05:33.709 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2023-12-04T21:05:33.709 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2023-12-04T21:05:33.768 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2023-12-04T21:05:33.768 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8" 2023-12-04T21:05:33.768 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:05:33.769 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:05:33.769 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2023-12-04T21:05:33.797 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:05:33.808 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:05:33.821 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:05:33.821 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:05:33.824 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:05:33.842 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:05:33.854 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:05:33.855 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:05:33.884 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:05:33.915 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:05:33.975 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:05:33.994 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:05:33.994 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:05:33.995 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:05:33.995 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:05:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:05:34.013 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-12-04T21:05:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:05:34.015 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-12-04T21:05:34.072 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:05:34.073 INFO:teuthology.task.kernel:Running kernel on smithi114: 4.18.0-489.el8.x86_64 2023-12-04T21:05:34.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2023-12-04T21:05:34.075 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:05:34.075 INFO:teuthology.task.kernel:Running kernel on smithi007: 4.18.0-489.el8.x86_64 2023-12-04T21:05:34.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2023-12-04T21:06:14.726 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - AppStream 921 kB/s | 34 MB 00:38 2023-12-04T21:07:01.319 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - BaseOS 1.4 MB/s | 55 MB 00:40 2023-12-04T21:08:26.488 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - AppStream 206 kB/s | 34 MB 02:49 2023-12-04T21:09:46.259 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras 116 B/s | 18 kB 02:37 2023-12-04T21:09:46.793 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-12-04T21:10:42.247 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:55 2023-12-04T21:10:42.248 INFO:teuthology.orchestra.run.smithi114.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:10:42.248 INFO:teuthology.orchestra.run.smithi114.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:10:42.248 INFO:teuthology.orchestra.run.smithi114.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:10:43.923 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 16 MB 00:01 2023-12-04T21:10:47.696 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 340 kB/s | 24 kB 00:00 2023-12-04T21:10:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:10:51.238 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:10:51.238 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:10:51.516 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-12-04T21:10:51.516 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-12-04T21:10:51.516 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout:Install 3 Packages 2023-12-04T21:10:51.517 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-12-04T21:10:51.518 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 90 M 2023-12-04T21:10:51.518 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 96 M 2023-12-04T21:10:51.518 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2023-12-04T21:11:23.934 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - BaseOS 329 kB/s | 55 MB 02:50 2023-12-04T21:11:58.994 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - Extras 681 B/s | 18 kB 00:26 2023-12-04T21:11:59.556 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-04T21:12:53.235 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:53 2023-12-04T21:12:53.235 INFO:teuthology.orchestra.run.smithi007.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:12:53.235 INFO:teuthology.orchestra.run.smithi007.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:12:53.235 INFO:teuthology.orchestra.run.smithi007.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:13:18.091 INFO:teuthology.orchestra.run.smithi114.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:13:18.091 INFO:teuthology.orchestra.run.smithi114.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:13:18.091 INFO:teuthology.orchestra.run.smithi114.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:13:20.505 INFO:teuthology.orchestra.run.smithi114.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 667 kB/s | 10 MB 00:15 2023-12-04T21:13:22.905 INFO:teuthology.orchestra.run.smithi114.stdout:(2/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.4 MB/s | 43 MB 00:17 2023-12-04T21:13:28.780 INFO:teuthology.orchestra.run.smithi114.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.5 MB/s | 36 MB 00:23 2023-12-04T21:13:28.780 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:13:28.780 INFO:teuthology.orchestra.run.smithi114.stdout:Total 583 kB/s | 90 MB 02:37 2023-12-04T21:13:29.454 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2023-12-04T21:13:29.888 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2023-12-04T21:13:29.888 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2023-12-04T21:13:31.123 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2023-12-04T21:13:31.164 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2023-12-04T21:13:33.938 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2023-12-04T21:13:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:38.029 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:38.428 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:41.851 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:42.110 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:14:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-12-04T21:14:05.795 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-12-04T21:14:05.942 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:14:06.157 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:14:06.157 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:14:06.157 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:14:06.158 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:14:06.158 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:14:06.158 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T21:14:06.158 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:14:06.158 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:14:06.367 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:14:06.367 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:14:06.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2023-12-04T21:14:06.401 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2023-12-04T21:14:06.401 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2023-12-04T21:14:06.401 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.02 2023-12-04T21:14:06.401 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 129.el8 2023-12-04T21:14:06.401 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2023-12-04T21:14:06.401 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Group : System Environment/Base 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 9265586 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:14:06.402 INFO:teuthology.orchestra.run.smithi114.stdout:Relocations : (not relocatable) 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : CentOS BuildSystem 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-12-04T21:14:06.403 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:14:06.404 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:14:06.404 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:14:06.404 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2023-12-04T21:14:06.404 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-12-04T21:14:06.404 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:14:06.406 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:14:06.406 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:14:06.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:14:07.708 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2023-12-04T21:14:10.401 INFO:teuthology.orchestra.run.smithi114.stderr:done 2023-12-04T21:14:10.402 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2023-12-04T21:14:10.435 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.Fq3qGAXOkm 2023-12-04T21:14:10.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Fq3qGAXOkm 2023-12-04T21:14:10.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.Fq3qGAXOkm 2023-12-04T21:14:10.646 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.Fq3qGAXOkm is 6KB 2023-12-04T21:14:10.696 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.Fq3qGAXOkm 2023-12-04T21:14:10.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:14:10.785 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:14:10.785 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:14:10.786 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:14:10.786 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:14:10.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:14:10.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2023-12-04T21:14:40.978 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T21:14:40.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-12-04T21:14:40.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:04.418 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2023-12-04T21:15:13.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-12-04T21:15:13.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:31.810 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2023-12-04T21:15:43.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-12-04T21:15:43.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:46.675 DEBUG:teuthology.orchestra.run.smithi114:> true 2023-12-04T21:15:47.042 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-12-04T21:15:47.042 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:15:47.042 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-12-04T21:15:47.061 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:15:47.061 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:15:47.061 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:15:47.062 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:15:48.062 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T21:15:48.062 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:15:48.134 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2023-12-04T21:15:48.150 DEBUG:teuthology.parallel:result is None 2023-12-04T21:17:29.202 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 59 kB/s | 16 MB 04:35 2023-12-04T21:17:33.234 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 402 kB/s | 24 kB 00:00 2023-12-04T21:17:33.452 INFO:teuthology.orchestra.run.smithi007.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:17:36.958 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:17:36.959 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:17:37.282 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-12-04T21:17:37.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-12-04T21:17:37.283 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-12-04T21:17:37.283 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-12-04T21:17:37.283 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-12-04T21:17:37.283 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout:Install 3 Packages 2023-12-04T21:17:37.284 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-04T21:17:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 90 M 2023-12-04T21:17:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 96 M 2023-12-04T21:17:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-12-04T21:17:50.608 INFO:teuthology.orchestra.run.smithi007.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:17:50.608 INFO:teuthology.orchestra.run.smithi007.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:17:50.608 INFO:teuthology.orchestra.run.smithi007.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:17:53.174 INFO:teuthology.orchestra.run.smithi007.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 658 kB/s | 10 MB 00:15 2023-12-04T21:17:55.391 INFO:teuthology.orchestra.run.smithi007.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.0 MB/s | 36 MB 00:17 2023-12-04T21:17:55.950 INFO:teuthology.orchestra.run.smithi007.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.3 MB/s | 43 MB 00:18 2023-12-04T21:17:55.950 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:17:55.950 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.8 MB/s | 90 MB 00:18 2023-12-04T21:17:56.659 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-12-04T21:17:57.103 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-12-04T21:17:57.103 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-12-04T21:17:58.398 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-12-04T21:17:58.442 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-12-04T21:18:01.391 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-12-04T21:18:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:05.618 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:06.016 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:18:09.691 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:18:09.966 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:33.493 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:33.519 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:33.879 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:33.879 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:33.879 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:18:34.683 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:34.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-04T21:18:34.684 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-12-04T21:18:34.684 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:18:34.684 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:18:34.684 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-04T21:18:34.684 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-12-04T21:18:34.819 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:18:35.046 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:18:35.047 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:18:35.047 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:18:35.047 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:18:35.047 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:18:35.047 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T21:18:35.047 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:18:35.048 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:18:35.266 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:18:35.266 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:18:35.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2023-12-04T21:18:35.300 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.02 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 129.el8 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Group : System Environment/Base 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 9265586 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:18:35.301 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Relocations : (not relocatable) 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : CentOS BuildSystem 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:18:35.302 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:18:35.303 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2023-12-04T21:18:35.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-04T21:18:35.303 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:18:35.304 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:18:35.304 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:18:35.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:18:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-12-04T21:18:38.672 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-12-04T21:18:38.673 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-12-04T21:18:38.693 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.Pe5DLZOupK 2023-12-04T21:18:38.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Pe5DLZOupK 2023-12-04T21:18:38.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.Pe5DLZOupK 2023-12-04T21:18:38.927 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.Pe5DLZOupK is 6KB 2023-12-04T21:18:38.976 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.Pe5DLZOupK 2023-12-04T21:18:38.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:18:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:18:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:18:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:18:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:18:39.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:18:39.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2023-12-04T21:19:09.218 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T21:19:09.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-04T21:19:09.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:19:27.586 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-12-04T21:19:36.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-04T21:19:36.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:20:36.591 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:20:48.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-04T21:20:48.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:20:53.943 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-12-04T21:20:54.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-04T21:20:54.234 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:20:54.234 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-12-04T21:20:54.256 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:20:54.256 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:20:54.256 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:20:54.256 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:20:55.257 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T21:20:55.258 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:20:55.341 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2023-12-04T21:20:55.355 DEBUG:teuthology.parallel:result is None 2023-12-04T21:20:55.355 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:20:55.365 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:20:55.365 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:20:55.367 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:20:55.391 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:20:55.402 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:20:55.413 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:20:55.413 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:20:55.430 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:20:55.483 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:20:55.494 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:20:55.495 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:20:55.497 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:20:55.534 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:55.545 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:55.546 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:55.556 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:55.557 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:20:55.569 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:20:55.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:20:55.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:20:55.622 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:20:55.634 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:20:55.634 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:20:55.664 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:20:55.684 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:00.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:00.839 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:05.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:05.928 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-04T21:21:05.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:21:06.003 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:11.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:11.098 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:13.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:13.144 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-12-04T21:21:13.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:21:13.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-12-04T21:21:13.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2023-12-04T21:21:13.298 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:21:13.332 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:21:13.653 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:21:13.661 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:21:13.661 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:21:13.671 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:21:13.678 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:21:13.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2023-12-04T21:21:13.785 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2023-12-04T21:21:13.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2023-12-04T21:21:13.879 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2023-12-04T21:21:13.880 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:21:13.881 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2023-12-04T21:21:13.944 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2023-12-04T21:21:13.944 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2023-12-04T21:21:13.966 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2023-12-04T21:21:13.966 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:21:13.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:21:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:21:14.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:21:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:21:14.036 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:21:14.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:21:14.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:21:14.104 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:21:14.122 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-04T21:21:14.122 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:21:14.138 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:21:14.139 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:37:05.472 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365/ansible_failures.yaml 2023-12-04T21:37:05.475 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:37:05.543 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9a80953861f34f7fb8dfef49ee26a401 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:37:05.544 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:37:05.552 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:37:05.552 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:37:05.561 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:37:10.679 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:37:10.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:37:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:37:10.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:37:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:37:10.835 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:37:10.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:37:10.857 INFO:teuthology.task.internal:Duration was 957.196347 seconds 2023-12-04T21:37:10.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:37:10.873 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:37:10.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:37:10.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:37:10.923 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:37:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:37:11.404 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:37:11.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-12-04T21:37:11.405 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:37:11.433 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2023-12-04T21:37:11.434 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:37:11.457 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:37:11.457 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:37:11.476 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:37:11.532 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:37:11.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:37:11.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:37:11.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:37:11.642 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:37:11.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:37:11.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:37:11.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:37:11.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:37:11.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:37:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-12-04T21:37:11.767 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2023-12-04T21:37:11.831 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:37:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:37:11.867 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:37:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:37:11.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:37:11.900 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:37:11.902 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365/remote/smithi007 2023-12-04T21:37:11.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:37:11.973 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365/remote/smithi114 2023-12-04T21:37:11.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:37:12.016 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:37:12.016 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:37:12.018 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:37:12.068 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:37:12.093 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:37:12.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:37:12.173 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:37:12.174 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:37:12.176 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:37:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:37 /home/ubuntu/cephtest 2023-12-04T21:37:12.196 INFO:teuthology.orchestra.run.smithi007.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:37 /home/ubuntu/cephtest 2023-12-04T21:37:12.198 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:37:12.209 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:37:12.239 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:37:17.275 DEBUG:teuthology.nuke:shortname: smithi007 2023-12-04T21:37:17.276 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:37:17.299 DEBUG:teuthology.nuke:shortname: smithi114 2023-12-04T21:37:17.299 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:37:17.302 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:34.516514', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMMwROLMYke0SBoH/oM1wcptn1WJqmMmR1hYKvOCw8NIK2fd2r9BPA+6HV+C2pH1mfoRNw2nqchMtSw/AN+ZsCk='} 2023-12-04T21:37:17.319 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477365', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:34.518582', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKL2lk7VvfQkRyVlMGdlm7qCGlg/26UOP/TwnUauVAyRrhQN+EdsBvFd9L+SJWgo8TPePobSuZtltKTbuolFqtg='} 2023-12-04T21:37:17.345 INFO:teuthology.orchestra.console.smithi007:Power off 2023-12-04T21:37:17.345 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:37:17.371 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2023-12-04T21:37:17.371 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:17.385 INFO:teuthology.orchestra.console.smithi114:Power off 2023-12-04T21:37:17.385 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:37:17.400 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-12-04T21:37:17.407 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2023-12-04T21:37:17.407 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:17.427 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-12-04T21:37:21.401 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:21.428 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:21.535 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-12-04T21:37:22.510 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:25.540 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:26.511 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:26.651 DEBUG:teuthology.orchestra.console.smithi114:check power output: 2023-12-04T21:37:27.622 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:30.653 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:31.625 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:31.763 DEBUG:teuthology.orchestra.console.smithi114:check power output: 2023-12-04T21:37:32.735 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:35.764 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:35.878 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2023-12-04T21:37:35.979 INFO:teuthology.orchestra.console.smithi114:Power off completed 2023-12-04T21:37:36.736 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:37.846 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:41.853 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:42.965 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:46.970 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:48.079 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:52.084 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:53.193 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:37:57.196 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:37:58.306 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:38:02.307 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:03.418 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2023-12-04T21:38:07.419 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:07.528 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2023-12-04T21:38:07.629 INFO:teuthology.orchestra.console.smithi007:Power off completed 2023-12-04T21:38:07.735 INFO:teuthology.run:Summary data: description: rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} duration: 957.1963472366333 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9a80953861f34f7fb8dfef49ee26a401 status: dead success: false 2023-12-04T21:38:07.736 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:38:07.809 INFO:teuthology.run:DEAD