2023-12-04T21:05:34.879 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:05:34.879 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370 --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370/orig.config.yaml 2023-12-04T21:05:34.921 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:05:44.863 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} email: null first_in_suite: false job_id: '7477370' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 rgw max chunk size: 4194304 rgw obj stripe size: 6291456 setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd heartbeat grace: 40 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 rgw override bucket index max shards: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) - \(OSD_SLOW_PING_TIME sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 openssl_keys: rgw.client.0: ca: root client: client.0 embed-key: true rgw.client.1: ca: root client: client.1 embed-key: true root: client: client.0 cn: teuthology install: - client.0 - client.1 key-type: rsa:4096 rgw: client.0: ssl certificate: rgw.client.0 client.1: ssl certificate: rgw.client.1 compression type: random ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: civetweb storage classes: LUKEWARM, FROZEN s3tests: slow_backend: true selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEw048L8LcPQRn11JpzmAWHxorU/lxKIR1TdTBt4sg5CQxFGDq8N4mdwTVAkd2z4dlIGUyh+oKCzk+QlYijbOQU= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2E8lc0yBJeGUOBl+LrPytzzfzBnxgRFB/TRzDOrvG5RU1DABu/0Ou7ngQtiPX30Hrv8+5pKd+ruDgvOQMxJ/Y= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_rgw_throttle.sh - rgw/test_librgw_file.sh - ragweed: client.0: default-branch: ceph-pacific rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-pacific force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: ceph-pacific rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:05:44.864 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:05:44.864 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:05:44.864 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:05:44.866 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:05:44.888 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:05:44.888 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:05:44.888 INFO:teuthology.packaging:ref: None 2023-12-04T21:05:44.888 INFO:teuthology.packaging:tag: None 2023-12-04T21:05:44.888 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:05:44.888 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:05:44.888 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:05:45.109 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:05:45.109 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:05:45.117 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:05:45.117 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:05:45.145 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:05:45.161 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:05:45.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:05:55.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:36.102592', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEw048L8LcPQRn11JpzmAWHxorU/lxKIR1TdTBt4sg5CQxFGDq8N4mdwTVAkd2z4dlIGUyh+oKCzk+QlYijbOQU='} 2023-12-04T21:06:00.221 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:36.101290', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2E8lc0yBJeGUOBl+LrPytzzfzBnxgRFB/TRzDOrvG5RU1DABu/0Ou7ngQtiPX30Hrv8+5pKd+ruDgvOQMxJ/Y='} 2023-12-04T21:06:00.221 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:06:15.554 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-04T21:06:15.554 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-04T21:06:15.554 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:06:20.644 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f491aeb7ee0>, signals=[15]) 2023-12-04T21:06:20.645 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:06:20.656 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:06:20.656 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-12-04T21:06:20.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:06:20.727 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2023-12-04T21:06:20.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:06:20.795 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:06:20.801 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-12-04T21:06:20.822 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-12-04T21:06:20.822 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-12-04T21:06:20.881 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2023-12-04T21:06:20.881 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8" 2023-12-04T21:06:20.881 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2023-12-04T21:06:20.881 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:06:20.882 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:06:20.883 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-12-04T21:06:20.907 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2023-12-04T21:06:20.927 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2023-12-04T21:06:20.927 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2023-12-04T21:06:20.985 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2023-12-04T21:06:20.985 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8" 2023-12-04T21:06:20.985 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:06:20.986 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:06:20.987 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2023-12-04T21:06:21.015 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:06:21.025 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:06:21.062 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:06:21.062 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:06:21.065 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:06:21.082 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:06:21.088 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:06:21.088 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:06:21.125 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:06:21.152 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:06:21.201 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:06:21.214 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:06:21.214 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:06:21.215 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:06:21.215 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:06:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:06:21.232 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2023-12-04T21:06:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:06:21.235 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-12-04T21:06:21.291 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:06:21.291 INFO:teuthology.task.kernel:Running kernel on smithi145: 4.18.0-489.el8.x86_64 2023-12-04T21:06:21.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2023-12-04T21:06:21.295 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:06:21.295 INFO:teuthology.task.kernel:Running kernel on smithi050: 4.18.0-489.el8.x86_64 2023-12-04T21:06:21.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2023-12-04T21:07:00.880 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - AppStream 950 kB/s | 34 MB 00:36 2023-12-04T21:07:46.335 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - BaseOS 1.4 MB/s | 55 MB 00:38 2023-12-04T21:08:21.365 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras 680 B/s | 18 kB 00:26 2023-12-04T21:08:21.915 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-12-04T21:09:12.561 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 8 - AppStream 208 kB/s | 34 MB 02:48 2023-12-04T21:09:14.785 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:52 2023-12-04T21:09:14.785 INFO:teuthology.orchestra.run.smithi050.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:09:14.786 INFO:teuthology.orchestra.run.smithi050.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:09:14.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:11:28.166 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 122 kB/s | 16 MB 02:13 2023-12-04T21:11:32.197 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 440 kB/s | 24 kB 00:00 2023-12-04T21:11:32.416 INFO:teuthology.orchestra.run.smithi050.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:11:35.938 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:11:35.938 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:11:36.251 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:11:36.252 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout:Install 3 Packages 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 90 M 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 96 M 2023-12-04T21:11:36.253 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-12-04T21:11:50.715 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:11:50.715 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:11:50.715 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:11:54.221 INFO:teuthology.orchestra.run.smithi050.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 581 kB/s | 10 MB 00:17 2023-12-04T21:11:56.629 INFO:teuthology.orchestra.run.smithi050.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.8 MB/s | 36 MB 00:20 2023-12-04T21:11:58.796 INFO:teuthology.orchestra.run.smithi050.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 1.9 MB/s | 43 MB 00:22 2023-12-04T21:11:58.796 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:11:58.796 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.0 MB/s | 90 MB 00:22 2023-12-04T21:11:59.504 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-12-04T21:12:00.002 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-12-04T21:12:00.003 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-12-04T21:12:01.307 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-12-04T21:12:01.338 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-12-04T21:12:04.323 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-12-04T21:12:04.725 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 8 - BaseOS 339 kB/s | 55 MB 02:45 2023-12-04T21:12:04.869 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:12:08.691 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:12:09.094 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:12:12.769 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:12:13.097 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:12:37.140 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:12:37.165 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:12:37.534 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:12:37.534 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:12:37.534 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-04T21:12:38.442 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-12-04T21:12:38.585 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:12:38.816 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:12:38.816 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:12:38.816 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:12:38.816 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:12:38.817 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:12:38.817 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T21:12:38.817 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:12:38.817 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:12:39.035 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:12:39.036 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:12:39.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.02 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 129.el8 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:12:39.071 INFO:teuthology.orchestra.run.smithi050.stdout:Group : System Environment/Base 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 9265586 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Relocations : (not relocatable) 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : CentOS BuildSystem 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2023-12-04T21:12:39.072 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-12-04T21:12:39.073 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:12:39.074 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:12:39.074 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:12:39.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:12:40.483 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-12-04T21:12:42.351 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-12-04T21:12:42.353 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-12-04T21:12:42.375 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.6R0o49P5VS 2023-12-04T21:12:42.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6R0o49P5VS 2023-12-04T21:12:42.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.6R0o49P5VS 2023-12-04T21:12:42.638 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.6R0o49P5VS is 6KB 2023-12-04T21:12:42.686 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.6R0o49P5VS 2023-12-04T21:12:42.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:12:42.774 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:12:42.775 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:12:42.775 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:12:42.775 INFO:teuthology.orchestra.run.smithi050.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:12:42.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:12:42.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2023-12-04T21:13:12.943 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T21:13:12.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-04T21:13:12.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:13:31.490 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-12-04T21:13:40.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-04T21:13:40.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:04.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-12-04T21:14:16.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-04T21:14:16.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:14:21.394 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-12-04T21:14:21.690 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-12-04T21:14:21.690 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:14:21.691 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-12-04T21:14:21.712 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:14:21.712 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:14:21.712 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:14:21.712 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:14:22.713 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T21:14:22.713 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:14:22.803 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-12-04T21:14:22.816 DEBUG:teuthology.parallel:result is None 2023-12-04T21:14:50.131 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 8 - Extras 116 B/s | 18 kB 02:37 2023-12-04T21:16:59.477 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 8 - Extras common packages 55 B/s | 6.9 kB 02:09 2023-12-04T21:17:55.143 INFO:teuthology.orchestra.run.smithi145.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:55 2023-12-04T21:17:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:17:55.144 INFO:teuthology.orchestra.run.smithi145.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:17:55.144 INFO:teuthology.orchestra.run.smithi145.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:21:32.642 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 75 kB/s | 16 MB 03:37 2023-12-04T21:21:36.433 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 408 kB/s | 24 kB 00:00 2023-12-04T21:21:36.637 INFO:teuthology.orchestra.run.smithi145.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:21:39.924 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:21:39.924 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:21:40.221 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2023-12-04T21:21:40.221 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-12-04T21:21:40.221 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout:Install 3 Packages 2023-12-04T21:21:40.222 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-12-04T21:21:40.223 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 90 M 2023-12-04T21:21:40.223 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 96 M 2023-12-04T21:21:40.223 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2023-12-04T21:24:05.591 INFO:teuthology.orchestra.run.smithi145.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:24:05.592 INFO:teuthology.orchestra.run.smithi145.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:24:05.592 INFO:teuthology.orchestra.run.smithi145.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:24:08.876 INFO:teuthology.orchestra.run.smithi145.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 631 kB/s | 10 MB 00:16 2023-12-04T21:24:11.349 INFO:teuthology.orchestra.run.smithi145.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.9 MB/s | 36 MB 00:18 2023-12-04T21:24:13.426 INFO:teuthology.orchestra.run.smithi145.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.1 MB/s | 43 MB 00:20 2023-12-04T21:24:13.427 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:24:13.427 INFO:teuthology.orchestra.run.smithi145.stdout:Total 598 kB/s | 90 MB 02:33 2023-12-04T21:24:14.099 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2023-12-04T21:24:14.559 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2023-12-04T21:24:14.560 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2023-12-04T21:24:15.807 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2023-12-04T21:24:15.837 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2023-12-04T21:24:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2023-12-04T21:24:19.180 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:24:22.775 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:24:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:24:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:24:26.942 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:24:48.785 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:24:48.808 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:24:49.183 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:24:49.183 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:24:49.184 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:24:49.926 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:24:49.927 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-12-04T21:24:49.927 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2023-12-04T21:24:49.927 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:24:49.927 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:24:49.927 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-12-04T21:24:49.927 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2023-12-04T21:24:50.106 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:24:50.323 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:24:50.323 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:24:50.323 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:24:50.323 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:24:50.323 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:24:50.324 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T21:24:50.324 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:24:50.324 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:24:50.532 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:24:50.533 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:24:50.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2023-12-04T21:24:50.566 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.02 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 129.el8 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Group : System Environment/Base 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 9265586 2023-12-04T21:24:50.567 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Relocations : (not relocatable) 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : CentOS BuildSystem 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:24:50.568 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2023-12-04T21:24:50.569 INFO:teuthology.orchestra.run.smithi145.stdout: 2023-12-04T21:24:50.569 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:24:50.569 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:24:50.570 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:24:50.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:24:52.131 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2023-12-04T21:24:54.066 INFO:teuthology.orchestra.run.smithi145.stderr:done 2023-12-04T21:24:54.067 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2023-12-04T21:24:54.103 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.rPwB0tUcSK 2023-12-04T21:24:54.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rPwB0tUcSK 2023-12-04T21:24:54.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.rPwB0tUcSK 2023-12-04T21:24:54.363 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.rPwB0tUcSK is 6KB 2023-12-04T21:24:54.412 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.rPwB0tUcSK 2023-12-04T21:24:54.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:24:54.503 INFO:teuthology.orchestra.run.smithi145.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:24:54.504 INFO:teuthology.orchestra.run.smithi145.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:24:54.504 INFO:teuthology.orchestra.run.smithi145.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:24:54.504 INFO:teuthology.orchestra.run.smithi145.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:24:54.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:24:54.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2023-12-04T21:25:24.674 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T21:25:24.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-12-04T21:25:24.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:25:48.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-12-04T21:25:57.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-12-04T21:25:57.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:26:57.007 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T21:27:09.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-12-04T21:27:09.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:27:14.356 DEBUG:teuthology.orchestra.run.smithi145:> true 2023-12-04T21:27:14.689 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-12-04T21:27:14.690 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:27:14.690 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2023-12-04T21:27:14.710 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:27:14.710 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:27:14.710 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:27:14.710 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:27:15.711 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T21:27:15.711 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:27:15.789 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2023-12-04T21:27:15.803 DEBUG:teuthology.parallel:result is None 2023-12-04T21:27:15.803 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:27:15.813 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:27:15.813 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:27:15.817 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:27:15.838 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:27:15.850 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:27:15.863 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:27:15.864 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:27:15.880 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:27:15.927 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:27:15.938 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:27:15.938 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:27:15.951 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:27:16.003 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:27:16.007 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:27:16.016 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:27:16.018 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:27:16.019 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:27:16.027 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:27:16.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:27:16.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:27:16.061 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:27:16.072 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:27:16.072 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:27:16.105 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:27:16.125 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:27:21.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:27:21.265 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:27:22.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:27:22.136 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-12-04T21:27:22.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:27:22.211 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:27:22.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:27:22.320 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:27:22.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:27:22.410 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2023-12-04T21:27:22.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:27:22.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-12-04T21:27:22.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2023-12-04T21:27:22.562 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:27:22.578 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:27:23.071 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:27:25.861 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:27:25.861 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:27:25.873 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:27:25.883 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:27:25.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2023-12-04T21:27:25.963 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2023-12-04T21:27:25.965 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2023-12-04T21:27:26.032 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2023-12-04T21:27:26.034 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:27:26.034 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2023-12-04T21:27:26.062 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2023-12-04T21:27:26.062 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2023-12-04T21:27:26.080 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2023-12-04T21:27:26.080 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:27:26.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:27:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:27:26.133 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:27:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:27:26.160 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:27:26.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:27:26.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:27:26.229 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:27:26.239 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T21:27:26.243 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T21:27:26.417 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:27:26.431 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:27:26.432 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:42:50.472 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370/ansible_failures.yaml 2023-12-04T21:42:50.474 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:42:50.544 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ea0d17ab821a42749ec3e0fa3b2951c0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:42:50.546 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:42:50.555 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:42:50.555 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:42:50.594 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:42:55.699 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:42:55.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:42:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:55.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:42:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:55.855 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:42:55.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:42:55.882 INFO:teuthology.task.internal:Duration was 930.020515 seconds 2023-12-04T21:42:55.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:42:55.896 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:42:55.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:42:55.899 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:42:55.945 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:42:55.954 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:42:56.268 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:42:56.268 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-12-04T21:42:56.268 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:42:56.296 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2023-12-04T21:42:56.296 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:42:56.319 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:42:56.320 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:42:56.339 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:42:56.396 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:42:56.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:42:56.423 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:42:56.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:42:59.549 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:42:59.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:42:59.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:42:59.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:42:59.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:59.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:59.662 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-12-04T21:42:59.687 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2023-12-04T21:42:59.718 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:59.781 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:42:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:42:59.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:42:59.811 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:42:59.813 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370/remote/smithi050 2023-12-04T21:42:59.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:42:59.889 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370/remote/smithi145 2023-12-04T21:42:59.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:42:59.942 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:42:59.943 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:42:59.945 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:42:59.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:43:00.006 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:43:00.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:43:00.020 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:43:00.020 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:43:00.023 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:43:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:42 /home/ubuntu/cephtest 2023-12-04T21:43:00.053 INFO:teuthology.orchestra.run.smithi145.stdout: 525354 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:42 /home/ubuntu/cephtest 2023-12-04T21:43:00.055 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:43:00.066 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:43:00.096 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:43:00.116 DEBUG:teuthology.nuke:shortname: smithi050 2023-12-04T21:43:00.117 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:43:00.141 DEBUG:teuthology.nuke:shortname: smithi145 2023-12-04T21:43:00.141 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:43:00.149 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:36.102592', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEw048L8LcPQRn11JpzmAWHxorU/lxKIR1TdTBt4sg5CQxFGDq8N4mdwTVAkd2z4dlIGUyh+oKCzk+QlYijbOQU='} 2023-12-04T21:43:00.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:59:36.101290', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2E8lc0yBJeGUOBl+LrPytzzfzBnxgRFB/TRzDOrvG5RU1DABu/0Ou7ngQtiPX30Hrv8+5pKd+ruDgvOQMxJ/Y='} 2023-12-04T21:43:00.195 INFO:teuthology.orchestra.console.smithi050:Power off 2023-12-04T21:43:00.196 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:43:00.209 INFO:teuthology.orchestra.console.smithi145:Power off 2023-12-04T21:43:00.210 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:43:00.220 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2023-12-04T21:43:00.220 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:00.233 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2023-12-04T21:43:00.234 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:00.242 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-12-04T21:43:00.255 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2023-12-04T21:43:04.244 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:04.256 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:05.355 DEBUG:teuthology.orchestra.console.smithi050:check power output: 2023-12-04T21:43:05.368 DEBUG:teuthology.orchestra.console.smithi145:check power output: 2023-12-04T21:43:09.357 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:09.369 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:10.470 DEBUG:teuthology.orchestra.console.smithi050:check power output: 2023-12-04T21:43:10.479 DEBUG:teuthology.orchestra.console.smithi145:check power output: 2023-12-04T21:43:14.471 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:14.481 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:14.583 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2023-12-04T21:43:14.685 INFO:teuthology.orchestra.console.smithi050:Power off completed 2023-12-04T21:43:15.591 DEBUG:teuthology.orchestra.console.smithi145:check power output: 2023-12-04T21:43:19.595 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:20.705 DEBUG:teuthology.orchestra.console.smithi145:check power output: 2023-12-04T21:43:24.710 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:25.822 DEBUG:teuthology.orchestra.console.smithi145:check power output: 2023-12-04T21:43:29.824 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:30.933 DEBUG:teuthology.orchestra.console.smithi145:check power output: 2023-12-04T21:43:34.934 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:43:35.046 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2023-12-04T21:43:35.147 INFO:teuthology.orchestra.console.smithi145:Power off completed 2023-12-04T21:43:35.253 INFO:teuthology.run:Summary data: description: rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} duration: 930.020515203476 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ea0d17ab821a42749ec3e0fa3b2951c0 status: dead success: false 2023-12-04T21:43:35.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:43:35.333 INFO:teuthology.run:DEAD