2023-12-04T21:05:14.005 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T21:05:14.005 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359 --owner scheduled_yuriw@teuthology --description rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} --name yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359/orig.config.yaml 2023-12-04T21:05:14.045 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:05:19.133 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359 branch: wip-yuri-testing-2023-11-27-1028-pacific description: rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} email: null first_in_suite: false job_id: '7477359' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-11-27-1028-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd heartbeat grace: 40 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) - \(OSD_SLOW_PING_TIME sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 openssl_keys: rgw.client.0: ca: root client: client.0 embed-key: true rgw.client.1: ca: root client: client.1 embed-key: true root: client: client.0 cn: teuthology install: - client.0 - client.1 key-type: rsa:4096 rgw: client.0: ssl certificate: rgw.client.0 client.1: ssl certificate: rgw.client.1 compression type: random ec-data-pool: true frontend: civetweb storage classes: LUKEWARM, FROZEN s3tests: slow_backend: true selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2023-11-27-1028-pacific sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4005 sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri-testing-2023-11-27-1028-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxBpKE+epeKcFzQn+DekqUwf0EigfG6cPJtZBMN1VPx55uUo0bISZbqOnlJ2SoVriV8UfmR10W/bdOxTGVuUYc= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2K0kM9Hh0ZvO0LxfLnZF5xNU59AOvhXqyhLkjqLUXk78Hdxd/4/B5YUQufZ63mn2x4NaxDqCMVPfzRURflhYw= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_rgw_throttle.sh - rgw/test_librgw_file.sh - ragweed: client.0: default-branch: ceph-pacific rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-pacific force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: ceph-pacific rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_20:56:20 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T21:05:19.133 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa; will attempt to use it 2023-12-04T21:05:19.134 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5474366c84b0c469f7b5285fcfee86232293d8b4/qa/tasks 2023-12-04T21:05:19.134 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T21:05:19.136 INFO:teuthology.task.internal:Checking packages... 2023-12-04T21:05:19.160 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5474366c84b0c469f7b5285fcfee86232293d8b4' 2023-12-04T21:05:19.160 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T21:05:19.160 INFO:teuthology.packaging:ref: None 2023-12-04T21:05:19.160 INFO:teuthology.packaging:tag: None 2023-12-04T21:05:19.160 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:05:19.160 INFO:teuthology.packaging:sha1: 5474366c84b0c469f7b5285fcfee86232293d8b4 2023-12-04T21:05:19.160 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-11-27-1028-pacific 2023-12-04T21:05:24.396 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-270.g5474366c 2023-12-04T21:05:24.396 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T21:05:24.403 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T21:05:24.403 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T21:05:24.475 INFO:teuthology.task.internal:Saving configuration 2023-12-04T21:05:24.490 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T21:05:24.497 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:05:33.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:04.563015', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxBpKE+epeKcFzQn+DekqUwf0EigfG6cPJtZBMN1VPx55uUo0bISZbqOnlJ2SoVriV8UfmR10W/bdOxTGVuUYc='} 2023-12-04T21:05:33.377 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:04.564535', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2K0kM9Hh0ZvO0LxfLnZF5xNU59AOvhXqyhLkjqLUXk78Hdxd/4/B5YUQufZ63mn2x4NaxDqCMVPfzRURflhYw='} 2023-12-04T21:05:33.378 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T21:05:33.386 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-04T21:05:33.386 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-04T21:05:33.386 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T21:05:33.453 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdbd09afee0>, signals=[15]) 2023-12-04T21:05:33.453 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T21:05:33.462 INFO:teuthology.task.internal:Opening connections... 2023-12-04T21:05:33.462 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-12-04T21:05:33.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:05:33.533 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2023-12-04T21:05:33.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:05:33.603 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T21:05:33.611 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2023-12-04T21:05:33.631 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2023-12-04T21:05:33.632 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2023-12-04T21:05:33.691 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2023-12-04T21:05:33.691 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8" 2023-12-04T21:05:33.691 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:05:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:05:33.693 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-12-04T21:05:33.713 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2023-12-04T21:05:33.733 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2023-12-04T21:05:33.733 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2023-12-04T21:05:33.792 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="8" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="8" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el8" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-04T21:05:33.793 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T21:05:33.793 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2023-12-04T21:05:33.813 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T21:05:33.832 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T21:05:33.842 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T21:05:33.842 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:05:33.845 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2023-12-04T21:05:33.863 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T21:05:33.872 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T21:05:33.873 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:05:33.905 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2023-12-04T21:05:33.936 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T21:05:33.988 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T21:05:34.002 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T21:05:34.003 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T21:05:34.003 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:05:34.003 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T21:05:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:05:34.021 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2023-12-04T21:05:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:05:34.022 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-12-04T21:05:34.080 INFO:teuthology.orchestra.run.smithi183.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:05:34.080 INFO:teuthology.task.kernel:Running kernel on smithi183: 4.18.0-489.el8.x86_64 2023-12-04T21:05:34.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2023-12-04T21:05:34.082 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-489.el8.x86_64 2023-12-04T21:05:34.082 INFO:teuthology.task.kernel:Running kernel on smithi002: 4.18.0-489.el8.x86_64 2023-12-04T21:05:34.082 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2023-12-04T21:06:16.176 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 8 - AppStream 887 kB/s | 34 MB 00:39 2023-12-04T21:06:18.609 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 8 - AppStream 835 kB/s | 34 MB 00:42 2023-12-04T21:07:25.025 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 8 - BaseOS 898 kB/s | 55 MB 01:02 2023-12-04T21:09:16.136 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 8 - BaseOS 329 kB/s | 55 MB 02:50 2023-12-04T21:09:54.156 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 8 - Extras 617 B/s | 18 kB 00:29 2023-12-04T21:09:54.673 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2023-12-04T21:10:11.666 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 8 - Extras 115 B/s | 18 kB 02:38 2023-12-04T21:10:47.145 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:52 2023-12-04T21:10:47.146 INFO:teuthology.orchestra.run.smithi002.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:10:47.146 INFO:teuthology.orchestra.run.smithi002.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:10:47.146 INFO:teuthology.orchestra.run.smithi002.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:10:48.585 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-12-04T21:10:52.618 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 468 kB/s | 24 kB 00:00 2023-12-04T21:10:52.838 INFO:teuthology.orchestra.run.smithi002.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:10:56.326 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:10:56.326 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:10:56.634 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-12-04T21:10:56.635 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-12-04T21:10:56.635 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2023-12-04T21:10:56.635 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-12-04T21:10:56.635 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-12-04T21:10:56.635 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout:Install 3 Packages 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-12-04T21:10:56.636 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 90 M 2023-12-04T21:10:56.637 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 96 M 2023-12-04T21:10:56.637 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-12-04T21:12:24.545 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 8 - Extras common packages 54 B/s | 6.9 kB 02:12 2023-12-04T21:13:18.132 INFO:teuthology.orchestra.run.smithi002.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:13:19.202 INFO:teuthology.orchestra.run.smithi183.stdout:Copr repo for python3-asyncssh owned by ceph 0.0 B/s | 0 B 00:54 2023-12-04T21:13:19.203 INFO:teuthology.orchestra.run.smithi183.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': 2023-12-04T21:13:19.203 INFO:teuthology.orchestra.run.smithi183.stderr: - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org] 2023-12-04T21:13:19.203 INFO:teuthology.orchestra.run.smithi183.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried 2023-12-04T21:13:22.892 INFO:teuthology.orchestra.run.smithi002.stdout:(1/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.2 MB/s | 36 MB 00:16 2023-12-04T21:13:22.892 INFO:teuthology.orchestra.run.smithi002.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:13:22.892 INFO:teuthology.orchestra.run.smithi002.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:13:25.499 INFO:teuthology.orchestra.run.smithi002.stdout:(2/3): kernel-4.18.0-526.el8.x86_64.rpm 548 kB/s | 10 MB 00:18 2023-12-04T21:13:27.291 INFO:teuthology.orchestra.run.smithi002.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.1 MB/s | 43 MB 00:20 2023-12-04T21:13:27.291 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:13:27.291 INFO:teuthology.orchestra.run.smithi002.stdout:Total 609 kB/s | 90 MB 02:30 2023-12-04T21:13:27.998 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-12-04T21:13:28.445 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-12-04T21:13:28.445 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-12-04T21:13:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-12-04T21:13:29.794 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-12-04T21:13:32.760 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-12-04T21:13:33.269 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:37.017 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:13:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:41.081 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:13:41.353 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:04.730 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:04.754 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:14:05.116 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-12-04T21:14:05.928 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-12-04T21:14:06.929 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:14:07.159 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:14:07.159 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:14:07.159 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:14:07.160 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:14:07.160 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:14:07.160 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T21:14:07.160 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:14:07.160 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:14:07.379 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:14:07.379 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:14:07.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2023-12-04T21:14:07.414 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2023-12-04T21:14:07.414 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2023-12-04T21:14:07.414 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.02 2023-12-04T21:14:07.414 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 129.el8 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Group : System Environment/Base 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 9265586 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Relocations : (not relocatable) 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : CentOS BuildSystem 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:14:07.415 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-12-04T21:14:07.416 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:14:07.417 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:14:07.417 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:14:07.417 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:14:08.417 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2023-12-04T21:14:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:done 2023-12-04T21:14:10.339 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-12-04T21:14:10.361 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.8rNyYsQZ6K 2023-12-04T21:14:10.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8rNyYsQZ6K 2023-12-04T21:14:10.433 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.8rNyYsQZ6K 2023-12-04T21:14:10.571 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.8rNyYsQZ6K is 6KB 2023-12-04T21:14:10.619 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.8rNyYsQZ6K 2023-12-04T21:14:10.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:14:10.703 INFO:teuthology.orchestra.run.smithi002.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:14:10.703 INFO:teuthology.orchestra.run.smithi002.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:14:10.703 INFO:teuthology.orchestra.run.smithi002.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:14:10.703 INFO:teuthology.orchestra.run.smithi002.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:14:10.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:14:10.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2023-12-04T21:14:40.861 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T21:14:40.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-12-04T21:14:40.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:04.418 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-12-04T21:15:13.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-12-04T21:15:13.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:29.803 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 125 kB/s | 16 MB 02:10 2023-12-04T21:15:31.810 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-12-04T21:15:33.609 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 545 kB/s | 24 kB 00:00 2023-12-04T21:15:33.818 INFO:teuthology.orchestra.run.smithi183.stderr:Ignoring repositories: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 2023-12-04T21:15:37.083 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-04T21:15:37.083 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-04T21:15:37.390 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout:Install 3 Packages 2023-12-04T21:15:37.391 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-12-04T21:15:37.392 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 90 M 2023-12-04T21:15:37.392 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 96 M 2023-12-04T21:15:37.392 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2023-12-04T21:15:43.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-12-04T21:15:43.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:15:45.901 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-12-04T21:15:46.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-12-04T21:15:46.228 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:15:46.228 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-12-04T21:15:46.250 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:15:46.250 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:15:46.250 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:15:46.250 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:15:47.251 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T21:15:47.252 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:15:47.397 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2023-12-04T21:15:47.414 DEBUG:teuthology.parallel:result is None 2023-12-04T21:17:59.086 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:17:59.087 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-core-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-core-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:17:59.087 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-modules-4.18.0-526.el8.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.chpc.utah.edu/pub/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-modules-4.18.0-526.el8.x86_64.rpm [Could not resolve host: mirror.chpc.utah.edu] 2023-12-04T21:18:01.720 INFO:teuthology.orchestra.run.smithi183.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 743 kB/s | 10 MB 00:13 2023-12-04T21:18:03.645 INFO:teuthology.orchestra.run.smithi183.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 2.3 MB/s | 36 MB 00:15 2023-12-04T21:18:07.278 INFO:teuthology.orchestra.run.smithi183.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 2.2 MB/s | 43 MB 00:19 2023-12-04T21:18:07.278 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2023-12-04T21:18:07.278 INFO:teuthology.orchestra.run.smithi183.stdout:Total 612 kB/s | 90 MB 02:29 2023-12-04T21:18:07.953 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2023-12-04T21:18:08.399 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2023-12-04T21:18:08.399 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2023-12-04T21:18:09.641 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2023-12-04T21:18:09.682 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2023-12-04T21:18:12.517 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2023-12-04T21:18:13.051 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:16.586 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:16.992 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:18:20.427 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:18:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:42.569 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:42.592 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-04T21:18:42.968 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-04T21:18:43.774 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-04T21:18:43.774 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-12-04T21:18:43.775 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2023-12-04T21:18:43.775 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-04T21:18:43.775 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-04T21:18:43.775 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-12-04T21:18:43.775 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2023-12-04T21:18:43.931 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:18:44.255 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:18:44.255 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:18:44.256 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-04T21:18:44.256 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T21:18:44.256 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T21:18:44.256 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-04T21:18:44.256 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T21:18:44.256 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T21:18:44.467 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-04T21:18:44.467 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-04T21:18:44.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.02 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 129.el8 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Group : System Environment/Base 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 9265586 2023-12-04T21:18:44.502 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Relocations : (not relocatable) 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : CentOS BuildSystem 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T21:18:44.503 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T21:18:44.504 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2023-12-04T21:18:44.504 INFO:teuthology.orchestra.run.smithi183.stdout: 2023-12-04T21:18:44.504 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T21:18:44.504 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T21:18:44.505 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 4.18.0-526.el8.x86_64 2023-12-04T21:18:44.505 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T21:18:45.941 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2023-12-04T21:18:48.239 INFO:teuthology.orchestra.run.smithi183.stderr:done 2023-12-04T21:18:48.244 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2023-12-04T21:18:48.272 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.yzlnAsJj2o 2023-12-04T21:18:48.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yzlnAsJj2o 2023-12-04T21:18:48.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.yzlnAsJj2o 2023-12-04T21:18:48.488 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.yzlnAsJj2o is 6KB 2023-12-04T21:18:48.537 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.yzlnAsJj2o 2023-12-04T21:18:48.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T21:18:48.619 INFO:teuthology.orchestra.run.smithi183.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-04T21:18:48.620 INFO:teuthology.orchestra.run.smithi183.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-04T21:18:48.620 INFO:teuthology.orchestra.run.smithi183.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-04T21:18:48.620 INFO:teuthology.orchestra.run.smithi183.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-04T21:18:48.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-04T21:18:48.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2023-12-04T21:19:18.798 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-04T21:19:18.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-12-04T21:19:18.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:19:42.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2023-12-04T21:19:51.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-12-04T21:19:51.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:20:19.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2023-12-04T21:20:31.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-12-04T21:20:31.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T21:20:37.272 DEBUG:teuthology.orchestra.run.smithi183:> true 2023-12-04T21:20:37.530 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2023-12-04T21:20:37.530 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-526.el8.x86_64"... 2023-12-04T21:20:37.530 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2023-12-04T21:20:37.591 INFO:teuthology.orchestra.run.smithi183.stdout:4.18.0-526.el8.x86_64 2023-12-04T21:20:37.591 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-04T21:20:37.591 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T21:20:37.591 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T21:20:38.592 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-04T21:20:38.592 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T21:20:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2023-12-04T21:20:38.684 DEBUG:teuthology.parallel:result is None 2023-12-04T21:20:38.685 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T21:20:38.692 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T21:20:38.692 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:20:38.696 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T21:20:38.718 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T21:20:38.725 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T21:20:38.738 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T21:20:38.739 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:20:38.761 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T21:20:38.824 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T21:20:38.833 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T21:20:38.833 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:20:38.836 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T21:20:38.872 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:38.883 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:38.884 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:38.897 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T21:20:38.898 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T21:20:38.905 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T21:20:38.905 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:20:38.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T21:20:38.955 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T21:20:38.963 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T21:20:38.963 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:20:38.982 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T21:20:39.014 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:20:40.860 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:20:40.908 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:20:40.958 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:20:41.006 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-12-04T21:20:41.007 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:20:41.081 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:20:41.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T21:20:41.185 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:20:41.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T21:20:41.318 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2023-12-04T21:20:41.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T21:20:41.387 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2023-12-04T21:20:41.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2023-12-04T21:20:41.465 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:20:41.499 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:20:41.927 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T21:20:41.936 INFO:teuthology.task.internal:Starting timer... 2023-12-04T21:20:41.936 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T21:20:41.952 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T21:20:41.968 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-04T21:20:41.968 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2023-12-04T21:20:42.061 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2023-12-04T21:20:42.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2023-12-04T21:20:42.162 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2023-12-04T21:20:42.163 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T21:20:42.163 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2023-12-04T21:20:42.191 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2023-12-04T21:20:42.191 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2023-12-04T21:20:42.215 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2023-12-04T21:20:42.215 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2023-12-04T21:20:42.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:20:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:20:42.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:20:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:20:42.295 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T21:20:42.295 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:20:42.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2023-12-04T21:20:42.366 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T21:20:42.377 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-04T21:20:42.382 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-04T21:20:42.542 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T21:20:42.557 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T21:20:42.558 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T21:39:02.592 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359/ansible_failures.yaml 2023-12-04T21:39:02.594 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:39:02.659 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=083873ac78b1487bac446fc5842e4a80 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-12-04T21:39:02.660 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T21:39:02.670 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T21:39:02.670 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T21:39:02.690 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:39:02.791 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T21:39:02.878 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:39:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:39:02.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T21:39:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:39:02.947 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T21:39:02.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T21:39:02.969 INFO:teuthology.task.internal:Duration was 1101.032709 seconds 2023-12-04T21:39:02.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T21:39:02.983 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T21:39:02.983 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:39:02.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T21:39:03.033 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:39:03.042 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T21:39:03.334 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T21:39:03.335 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2023-12-04T21:39:03.335 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:39:03.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2023-12-04T21:39:03.363 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T21:39:03.385 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T21:39:03.385 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:39:03.406 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T21:39:03.461 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T21:39:03.461 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:39:03.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T21:39:03.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T21:39:03.570 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T21:39:03.570 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:39:03.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T21:39:03.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T21:39:03.640 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:39:03.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:39:03.677 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2023-12-04T21:39:03.696 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2023-12-04T21:39:03.722 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:39:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:39:03.768 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T21:39:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T21:39:03.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T21:39:03.796 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T21:39:03.797 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359/remote/smithi002 2023-12-04T21:39:03.798 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:39:03.859 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359/remote/smithi183 2023-12-04T21:39:03.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T21:39:03.924 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T21:39:03.924 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:39:03.927 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T21:39:03.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T21:39:03.959 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T21:39:03.959 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T21:39:03.972 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T21:39:03.973 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:39:03.990 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T21:39:04.007 INFO:teuthology.orchestra.run.smithi183.stdout: 525355 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:39 /home/ubuntu/cephtest 2023-12-04T21:39:04.009 INFO:teuthology.orchestra.run.smithi002.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 21:39 /home/ubuntu/cephtest 2023-12-04T21:39:04.011 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T21:39:04.022 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T21:39:04.049 INFO:teuthology.nuke:Checking targets against current locks 2023-12-04T21:39:04.075 DEBUG:teuthology.nuke:shortname: smithi002 2023-12-04T21:39:04.075 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:39:04.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:04.563015', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxBpKE+epeKcFzQn+DekqUwf0EigfG6cPJtZBMN1VPx55uUo0bISZbqOnlJ2SoVriV8UfmR10W/bdOxTGVuUYc='} 2023-12-04T21:39:04.104 DEBUG:teuthology.nuke:shortname: smithi183 2023-12-04T21:39:04.104 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T21:39:04.123 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-04_20:56:20-rgw-wip-yuri-testing-2023-11-27-1028-pacific-distro-default-smithi/7477359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 20:58:04.564535', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2K0kM9Hh0ZvO0LxfLnZF5xNU59AOvhXqyhLkjqLUXk78Hdxd/4/B5YUQufZ63mn2x4NaxDqCMVPfzRURflhYw='} 2023-12-04T21:39:04.153 INFO:teuthology.orchestra.console.smithi002:Power off 2023-12-04T21:39:04.154 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:39:04.176 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2023-12-04T21:39:04.177 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:04.192 INFO:teuthology.orchestra.console.smithi183:Power off 2023-12-04T21:39:04.193 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-04T21:39:04.201 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-12-04T21:39:04.213 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2023-12-04T21:39:04.214 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:04.238 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-12-04T21:39:08.202 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:08.239 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:08.312 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-12-04T21:39:08.347 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-12-04T21:39:12.317 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:12.347 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:12.427 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2023-12-04T21:39:12.455 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2023-12-04T21:39:12.530 INFO:teuthology.orchestra.console.smithi002:Power off completed 2023-12-04T21:39:16.460 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-04T21:39:16.570 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2023-12-04T21:39:16.671 INFO:teuthology.orchestra.console.smithi183:Power off completed 2023-12-04T21:39:16.777 INFO:teuthology.run:Summary data: description: rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} duration: 1101.032709121704 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=083873ac78b1487bac446fc5842e4a80 status: dead success: false 2023-12-04T21:39:16.777 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T21:39:16.847 INFO:teuthology.run:DEAD