2023-12-04T21:04:26.223 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:04:26.224 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354 --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi --description rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354/orig.config.yaml 2023-12-04T21:04:26.264 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:04:31.369 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} email: null first_in_suite: false job_id: '7477354' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 lockdep: true osd: debug ms: 1 debug osd: 20 lockdep: true osd objectstore: filestore osd shutdown pgref assert: true osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) - \(OSD_SLOW_PING_TIME sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 rgw: compression type: random ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: beast storage classes: LUKEWARM, FROZEN s3tests: slow_backend: true selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnwNMg1EkArIJ0yBbXXr0YVA/fo/5LS3TqzhjMgu6dq0TOHb5Hp5TJXCaNlXPtvNFiI2IBWH3NQbX1uWAJaHHA= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBZnHfsecstx7IldOBjTNY5m5DaVADqYOn+FjzjKJsp8/BBB6Lo82bFgtBxQm32xxkRjOz2Rq550OonV4OivYE= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_rgw_throttle.sh - rgw/test_librgw_file.sh - ragweed: client.0: default-branch: ceph-pacific rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-pacific force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: ceph-pacific rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:04:31.369 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:04:31.370 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:04:31.370 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:04:31.372 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:04:31.393 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:04:31.393 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:04:31.393 INFO:teuthology.packaging:ref: None 2023-12-04T21:04:31.393 INFO:teuthology.packaging:tag: None 2023-12-04T21:04:31.393 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:04:31.393 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:04:31.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:04:36.618 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:04:36.618 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:04:36.625 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:04:36.625 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:04:36.633 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:04:36.648 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:04:36.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:04:41.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:57:41.884070', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnwNMg1EkArIJ0yBbXXr0YVA/fo/5LS3TqzhjMgu6dq0TOHb5Hp5TJXCaNlXPtvNFiI2IBWH3NQbX1uWAJaHHA='} 2023-12-04T21:04:46.699 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:57:41.885583', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBZnHfsecstx7IldOBjTNY5m5DaVADqYOn+FjzjKJsp8/BBB6Lo82bFgtBxQm32xxkRjOz2Rq550OonV4OivYE='} 2023-12-04T21:04:46.699 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:04:46.714 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-04T21:04:46.714 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-04T21:04:46.714 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:04:46.799 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c36ca5ee0>, signals=[15]) 2023-12-04T21:04:46.799 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:04:46.816 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:04:46.816 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2023-12-04T21:04:46.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:04:46.888 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2023-12-04T21:04:46.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:04:46.956 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:04:46.965 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2023-12-04T21:04:46.985 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2023-12-04T21:04:46.986 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2023-12-04T21:04:47.046 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2023-12-04T21:04:47.046 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="8" 2023-12-04T21:04:47.046 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2023-12-04T21:04:47.046 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:04:47.046 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="8" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:04:47.047 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:04:47.047 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2023-12-04T21:04:52.102 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2023-12-04T21:04:52.149 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2023-12-04T21:04:52.149 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:04:52.210 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:04:52.211 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:04:52.211 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:04:52.211 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2023-12-04T21:04:57.110 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:04:57.127 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:04:57.136 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:04:57.136 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:04:57.139 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:04:57.160 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:04:57.170 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:04:57.171 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:04:57.201 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:04:57.229 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:05:07.686 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:05:07.704 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:05:07.704 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:05:07.704 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:05:07.705 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:05:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:05:07.727 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2023-12-04T21:05:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:05:07.729 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2023-12-04T21:05:07.788 INFO:teuthology.orchestra.run.smithi130.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:05:07.788 INFO:teuthology.task.kernel:Running kernel on smithi130: 4.18.0-489.el8.x86_64 2023-12-04T21:05:07.788 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2023-12-04T21:05:07.792 INFO:teuthology.orchestra.run.smithi044.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:05:07.792 INFO:teuthology.task.kernel:Running kernel on smithi044: 4.18.0-489.el8.x86_64 2023-12-04T21:05:07.792 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2023-12-04T21:05:47.131 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 8 - AppStream 956 kB/s | 34 MB 00:36 2023-12-04T21:06:32.301 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 8 - BaseOS 1.4 MB/s | 55 MB 00:38 2023-12-04T21:07:54.897 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 8 - AppStream 213 kB/s | 34 MB 02:44 2023-12-04T21:08:42.176 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 8 - BaseOS 1.4 MB/s | 55 MB 00:40 2023-12-04T21:09:15.946 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 8 - Extras 117 B/s | 18 kB 02:35 2023-12-04T21:09:16.166 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 8 - Extras 711 B/s | 18 kB 00:25 2023-12-04T21:09:16.455 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-12-04T21:10:08.110 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:51 2023-12-04T21:10:08.111 INFO:teuthology.orchestra.run.smithi130.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:10:08.111 INFO:teuthology.orchestra.run.smithi130.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:10:08.111 INFO:teuthology.orchestra.run.smithi130.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:11:26.943 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 8 - Extras common packages 54 B/s | 6.9 kB 02:10 2023-12-04T21:12:19.016 INFO:teuthology.orchestra.run.smithi044.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:51 2023-12-04T21:12:19.016 INFO:teuthology.orchestra.run.smithi044.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:12:19.017 INFO:teuthology.orchestra.run.smithi044.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:12:19.017 INFO:teuthology.orchestra.run.smithi044.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:13:40.356 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 201 kB/s | 16 MB 01:21 2023-12-04T21:13:44.365 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 424 kB/s | 24 kB 00:00 2023-12-04T21:13:44.584 INFO:teuthology.orchestra.run.smithi044.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:13:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:13:48.064 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:13:48.375 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2023-12-04T21:13:48.375 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2023-12-04T21:13:48.375 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2023-12-04T21:13:48.375 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2023-12-04T21:13:48.375 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2023-12-04T21:13:48.375 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout:Install 3 Packages 2023-12-04T21:13:48.376 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-12-04T21:13:48.377 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 90 M 2023-12-04T21:13:48.377 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 96 M 2023-12-04T21:13:48.377 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2023-12-04T21:16:10.442 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:16:10.442 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:16:10.442 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:16:13.875 INFO:teuthology.orchestra.run.smithi044.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 626 kB/s | 10 MB 00:16 2023-12-04T21:16:16.541 INFO:teuthology.orchestra.run.smithi044.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.9 MB/s | 36 MB 00:19 2023-12-04T21:16:17.350 INFO:teuthology.orchestra.run.smithi044.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.2 MB/s | 43 MB 00:19 2023-12-04T21:16:17.350 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:16:17.350 INFO:teuthology.orchestra.run.smithi044.stdout:Total 615 kB/s | 90 MB 02:28 2023-12-04T21:16:18.059 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2023-12-04T21:16:18.526 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2023-12-04T21:16:18.526 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2023-12-04T21:16:19.832 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2023-12-04T21:16:19.871 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2023-12-04T21:16:22.877 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2023-12-04T21:16:23.448 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:16:27.184 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:16:27.581 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:16:31.253 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:16:31.548 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:16:54.999 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:16:55.022 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:16:55.390 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:16:55.390 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:16:55.390 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:16:56.627 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:16:56.627 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-12-04T21:16:56.627 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2023-12-04T21:16:56.628 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:16:56.628 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:16:56.628 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-12-04T21:16:56.628 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2023-12-04T21:16:56.782 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:16:57.012 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:16:57.012 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:16:57.013 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:16:57.013 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:16:57.013 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:16:57.013 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T21:16:57.013 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:16:57.013 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:16:57.231 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:16:57.231 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:16:57.232 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.02 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 129.el8 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:16:57.266 INFO:teuthology.orchestra.run.smithi044.stdout:Group : System Environment/Base 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 9265586 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Relocations : (not relocatable) 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : CentOS BuildSystem 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:16:57.267 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:16:57.268 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2023-12-04T21:16:57.268 INFO:teuthology.orchestra.run.smithi044.stdout: 2023-12-04T21:16:57.268 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:16:57.268 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:16:57.268 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:16:57.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:16:58.608 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2023-12-04T21:17:00.852 INFO:teuthology.orchestra.run.smithi044.stderr:done 2023-12-04T21:17:00.855 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2023-12-04T21:17:00.888 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.aSMaT0OBLY 2023-12-04T21:17:00.888 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aSMaT0OBLY 2023-12-04T21:17:00.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.aSMaT0OBLY 2023-12-04T21:17:01.127 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.aSMaT0OBLY is 6KB 2023-12-04T21:17:01.176 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.aSMaT0OBLY 2023-12-04T21:17:01.194 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:17:01.260 INFO:teuthology.orchestra.run.smithi044.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:17:01.260 INFO:teuthology.orchestra.run.smithi044.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:17:01.260 INFO:teuthology.orchestra.run.smithi044.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:17:01.260 INFO:teuthology.orchestra.run.smithi044.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:17:01.261 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:17:01.413 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2023-12-04T21:17:31.416 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T21:17:31.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-12-04T21:17:31.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:17:49.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.44 2023-12-04T21:17:58.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-12-04T21:17:58.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:18:47.955 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 31 kB/s | 16 MB 08:39 2023-12-04T21:18:51.746 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 419 kB/s | 24 kB 00:00 2023-12-04T21:18:51.952 INFO:teuthology.orchestra.run.smithi130.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:18:55.322 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:18:55.322 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:18:55.600 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2023-12-04T21:18:55.601 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:18:55.602 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:18:55.602 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-12-04T21:18:55.602 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-12-04T21:18:55.602 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-12-04T21:18:55.602 INFO:teuthology.orchestra.run.smithi130.stdout:Install 3 Packages 2023-12-04T21:18:55.602 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-12-04T21:18:55.603 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 90 M 2023-12-04T21:18:55.603 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 96 M 2023-12-04T21:18:55.603 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2023-12-04T21:18:58.798 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:19:08.878 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:19:08.879 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:19:08.879 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:19:10.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-12-04T21:19:10.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:19:11.094 DEBUG:teuthology.orchestra.run.smithi044:> true 2023-12-04T21:19:11.380 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2023-12-04T21:19:11.380 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:19:11.380 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2023-12-04T21:19:11.388 INFO:teuthology.orchestra.run.smithi130.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 663 kB/s | 10 MB 00:15 2023-12-04T21:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:19:11.401 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:19:11.402 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:19:11.402 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:19:12.402 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T21:19:12.403 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:19:12.466 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2023-12-04T21:19:12.483 DEBUG:teuthology.parallel:result is None 2023-12-04T21:19:13.238 INFO:teuthology.orchestra.run.smithi130.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.1 MB/s | 36 MB 00:17 2023-12-04T21:19:13.647 INFO:teuthology.orchestra.run.smithi130.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.4 MB/s | 43 MB 00:17 2023-12-04T21:19:13.647 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:19:13.647 INFO:teuthology.orchestra.run.smithi130.stdout:Total 5.0 MB/s | 90 MB 00:18 2023-12-04T21:19:14.325 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2023-12-04T21:19:14.759 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2023-12-04T21:19:14.759 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2023-12-04T21:19:16.023 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2023-12-04T21:19:16.061 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2023-12-04T21:19:18.841 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2023-12-04T21:19:19.439 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:19:23.013 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:19:23.413 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:19:26.858 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:19:27.186 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:49.104 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:49.127 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:49.508 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:49.509 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:19:49.509 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:19:50.301 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:19:50.301 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-12-04T21:19:50.301 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2023-12-04T21:19:50.301 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:19:50.301 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:19:50.302 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-12-04T21:19:50.302 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-12-04T21:19:51.091 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:19:51.310 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:19:51.310 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:19:51.310 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:19:51.311 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:19:51.311 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:19:51.311 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T21:19:51.311 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:19:51.311 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:19:51.520 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:19:51.520 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:19:51.520 DEBUG:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.02 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 129.el8 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Group : System Environment/Base 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 9265586 2023-12-04T21:19:51.554 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Relocations : (not relocatable) 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : CentOS BuildSystem 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:19:51.555 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:19:51.556 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:19:51.556 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2023-12-04T21:19:51.556 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-12-04T21:19:51.556 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:19:51.556 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:19:51.557 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:19:51.557 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:19:52.985 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2023-12-04T21:19:54.903 INFO:teuthology.orchestra.run.smithi130.stderr:done 2023-12-04T21:19:54.904 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2023-12-04T21:19:54.937 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.Q9YOVgSQH7 2023-12-04T21:19:54.937 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Q9YOVgSQH7 2023-12-04T21:19:55.018 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.Q9YOVgSQH7 2023-12-04T21:19:55.188 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.Q9YOVgSQH7 is 6KB 2023-12-04T21:19:55.238 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.Q9YOVgSQH7 2023-12-04T21:19:55.256 DEBUG:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:19:55.333 INFO:teuthology.orchestra.run.smithi130.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:19:55.333 INFO:teuthology.orchestra.run.smithi130.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:19:55.333 INFO:teuthology.orchestra.run.smithi130.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:19:55.333 INFO:teuthology.orchestra.run.smithi130.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:19:55.334 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:19:55.494 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2023-12-04T21:20:25.497 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T21:20:25.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-12-04T21:20:25.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:21:25.501 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:21:34.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-12-04T21:21:34.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:21:34.918 DEBUG:teuthology.orchestra.run.smithi130:> true 2023-12-04T21:21:35.229 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-12-04T21:21:35.229 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:21:35.229 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2023-12-04T21:21:35.246 INFO:teuthology.orchestra.run.smithi130.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:21:35.247 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:21:35.247 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:21:35.247 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:21:36.249 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T21:21:36.249 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:21:36.360 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2023-12-04T21:21:36.377 DEBUG:teuthology.parallel:result is None 2023-12-04T21:21:36.377 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:21:36.384 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:21:36.384 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:21:36.387 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:21:36.408 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:21:36.415 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:21:36.424 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:21:36.424 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:21:36.451 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:21:36.487 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:21:36.493 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:21:36.494 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:21:36.524 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:21:36.561 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:21:36.570 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:21:36.571 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:21:36.582 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:21:36.584 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:21:36.591 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:21:36.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:21:36.593 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:21:36.635 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:21:36.644 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:21:36.644 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:21:36.667 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:21:36.689 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:37.819 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:37.863 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:37.912 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:37.962 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2023-12-04T21:21:37.962 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:21:38.036 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:38.102 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:21:38.173 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:38.223 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:21:38.770 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-12-04T21:21:38.770 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:21:39.286 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2023-12-04T21:21:39.289 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2023-12-04T21:21:39.370 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:21:39.725 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:21:40.111 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:21:40.118 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:21:40.119 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:21:40.133 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:21:40.140 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:21:40.141 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2023-12-04T21:21:40.227 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2023-12-04T21:21:40.229 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2023-12-04T21:21:40.413 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2023-12-04T21:21:40.415 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:21:40.415 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2023-12-04T21:21:40.451 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2023-12-04T21:21:40.451 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2023-12-04T21:21:40.479 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2023-12-04T21:21:40.479 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:21:40.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:21:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:21:40.526 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:21:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:21:40.552 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:21:40.553 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:21:40.601 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:21:40.662 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:21:40.672 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-04T21:21:40.673 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:21:40.688 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:21:40.689 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:38:50.188 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354/ansible_failures.yaml 2023-12-04T21:38:50.190 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:38:50.256 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d283030fda5c414caf7e442bee06eb58 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:38:50.258 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:38:50.267 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:38:50.267 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:38:50.276 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:38:50.374 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:38:50.463 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:38:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:50.501 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:38:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:50.533 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:38:50.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:38:50.560 INFO:teuthology.task.internal:Duration was 1030.441595 seconds 2023-12-04T21:38:50.560 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:38:50.570 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:38:50.570 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:38:50.573 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:38:50.620 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:38:50.629 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:38:51.039 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:38:51.040 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2023-12-04T21:38:51.040 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:38:51.069 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2023-12-04T21:38:51.069 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:38:51.091 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:38:51.091 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:38:51.112 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:38:51.166 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:38:51.166 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:38:51.197 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:38:51.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:38:51.284 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:38:51.284 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:38:51.317 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:38:51.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:38:51.355 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:51.359 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:51.391 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2023-12-04T21:38:51.410 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2023-12-04T21:38:51.440 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:51.503 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:38:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:38:51.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:38:51.535 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:38:51.537 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354/remote/smithi044 2023-12-04T21:38:51.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:38:51.606 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354/remote/smithi130 2023-12-04T21:38:51.607 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:38:51.662 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:38:51.662 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:38:51.665 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:38:51.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:38:51.725 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:38:51.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:38:51.735 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:38:51.735 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:38:51.738 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:38:51.755 INFO:teuthology.orchestra.run.smithi130.stdout: 525841 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:38 /home/ubuntu/cephtest 2023-12-04T21:38:51.757 INFO:teuthology.orchestra.run.smithi044.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:38 /home/ubuntu/cephtest 2023-12-04T21:38:51.759 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:38:51.767 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:38:51.787 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:38:51.813 DEBUG:teuthology.nuke:shortname: smithi044 2023-12-04T21:38:51.813 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:38:51.836 DEBUG:teuthology.nuke:shortname: smithi130 2023-12-04T21:38:51.836 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:38:51.839 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:57:41.884070', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnwNMg1EkArIJ0yBbXXr0YVA/fo/5LS3TqzhjMgu6dq0TOHb5Hp5TJXCaNlXPtvNFiI2IBWH3NQbX1uWAJaHHA='} 2023-12-04T21:38:51.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:57:41.885583', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBZnHfsecstx7IldOBjTNY5m5DaVADqYOn+FjzjKJsp8/BBB6Lo82bFgtBxQm32xxkRjOz2Rq550OonV4OivYE='} 2023-12-04T21:38:51.889 INFO:teuthology.orchestra.console.smithi044:Power off 2023-12-04T21:38:51.890 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:38:51.907 INFO:teuthology.orchestra.console.smithi130:Power off 2023-12-04T21:38:51.907 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:38:51.916 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2023-12-04T21:38:51.916 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:51.929 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2023-12-04T21:38:51.929 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:51.938 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2023-12-04T21:38:51.951 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2023-12-04T21:38:55.939 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:55.951 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:38:56.048 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2023-12-04T21:38:56.059 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2023-12-04T21:39:00.053 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:00.061 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:00.163 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2023-12-04T21:39:00.169 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2023-12-04T21:39:04.167 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:04.174 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:04.277 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2023-12-04T21:39:04.285 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2023-12-04T21:39:04.378 INFO:teuthology.orchestra.console.smithi044:Power off completed 2023-12-04T21:39:04.386 INFO:teuthology.orchestra.console.smithi130:Power off completed 2023-12-04T21:39:04.492 INFO:teuthology.run:Summary data: description: rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} duration: 1030.4415950775146 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d283030fda5c414caf7e442bee06eb58 status: dead success: false 2023-12-04T21:39:04.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:39:04.568 INFO:teuthology.run:DEAD