2022-11-28T16:38:03.472 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-11-28T16:38:03.475 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734 --description rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734/orig.config.yaml 2022-11-28T16:38:03.534 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-11-28T16:38:03.617 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734 branch: wip-yuri7-testing-2022-11-18-1548 description: rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} email: ceph-qa@ceph.io first_in_suite: false job_id: '7090734' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri7-testing-2022-11-18-1548 ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 1 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd pool default size: 2 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon max pg per osd: 2 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max pg per osd hard ratio: 1 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random create_rbd_pool: false flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(TOO_FEW_PGS\) - \(PENDING_CREATING_PGS\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) mon_bind_msgr2: false pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri7-testing-2022-11-18-1548 sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri7-testing-2022-11-18-1548 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 targets: smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjLTrhlQv/MRDlz/xDR+KfvwQGplvud63jKhW4rBQhgjkX1IfQanNVyayVlR8c8VFXyxLPGmBARRIjD9A16Edo= tasks: - install: null - ceph: null - osd_max_pg_per_osd: pg_num: 2 test_create_from_mon: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4da97cf64e542f347ec47b7bdbe5eca99759f9b7 timestamp: 2022-11-23_21:46:38 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23126 2022-11-28T16:38:03.617 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa; will attempt to use it 2022-11-28T16:38:03.618 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks 2022-11-28T16:38:03.619 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-11-28T16:38:03.621 INFO:teuthology.task.internal:Checking packages... 2022-11-28T16:38:03.648 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '48d203d9b836cc224eb4085f6198d948c6fa1bb1' 2022-11-28T16:38:03.648 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-11-28T16:38:03.649 INFO:teuthology.packaging:ref: None 2022-11-28T16:38:03.649 INFO:teuthology.packaging:tag: None 2022-11-28T16:38:03.650 INFO:teuthology.packaging:branch: wip-yuri7-testing-2022-11-18-1548 2022-11-28T16:38:03.650 INFO:teuthology.packaging:sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 2022-11-28T16:38:03.651 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2022-11-18-1548 2022-11-28T16:38:03.816 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-872.g48d203d9 2022-11-28T16:38:03.816 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-11-28T16:38:03.825 INFO:teuthology.task.internal:no buildpackages task found 2022-11-28T16:38:03.826 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-11-28T16:38:03.834 INFO:teuthology.task.internal:Saving configuration 2022-11-28T16:38:03.847 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-11-28T16:38:03.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-11-28T16:38:03.877 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-11-28 16:33:00.236648', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjLTrhlQv/MRDlz/xDR+KfvwQGplvud63jKhW4rBQhgjkX1IfQanNVyayVlR8c8VFXyxLPGmBARRIjD9A16Edo='} 2022-11-28T16:38:03.878 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-11-28T16:38:03.884 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1'] 2022-11-28T16:38:03.885 INFO:teuthology.run_tasks:Running task console_log... 2022-11-28T16:38:03.936 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdbff504378>, signals=[15]) 2022-11-28T16:38:03.938 INFO:teuthology.run_tasks:Running task internal.connect... 2022-11-28T16:38:03.946 INFO:teuthology.task.internal:Opening connections... 2022-11-28T16:38:03.947 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2022-11-28T16:38:03.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-28T16:38:04.086 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-11-28T16:38:04.094 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2022-11-28T16:38:04.115 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2022-11-28T16:38:04.116 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2022-11-28T16:38:04.173 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2022-11-28T16:38:04.174 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8" 2022-11-28T16:38:04.174 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2022-11-28T16:38:04.174 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2022-11-28T16:38:04.174 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8" 2022-11-28T16:38:04.175 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8" 2022-11-28T16:38:04.175 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 8" 2022-11-28T16:38:04.175 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2022-11-28T16:38:04.175 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-11-28T16:38:04.176 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2022-11-28T16:38:04.176 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-11-28T16:38:04.176 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-11-28T16:38:04.176 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-11-28T16:38:04.179 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2022-11-28T16:38:04.201 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-11-28T16:38:04.211 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-11-28T16:38:04.220 INFO:teuthology.task.internal:Checking for old test directory... 2022-11-28T16:38:04.220 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2022-11-28T16:38:04.239 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-11-28T16:38:04.247 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-11-28T16:38:04.248 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2022-11-28T16:38:04.321 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-11-28T16:38:04.361 INFO:teuthology.run_tasks:Running task kernel... 2022-11-28T16:38:04.380 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-11-28T16:38:04.382 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-11-28T16:38:04.383 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2022-11-28T16:38:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T16:38:04.407 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2022-11-28T16:38:04.468 INFO:teuthology.orchestra.run.smithi161.stdout:4.18.0-408.el8.x86_64 2022-11-28T16:38:04.468 INFO:teuthology.task.kernel:Running kernel on smithi161: 4.18.0-408.el8.x86_64 2022-11-28T16:38:04.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2022-11-28T16:38:07.402 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - AppStream 40 MB/s | 26 MB 00:00 2022-11-28T16:38:12.636 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - BaseOS 45 MB/s | 26 MB 00:00 2022-11-28T16:38:16.661 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - Extras 123 kB/s | 18 kB 00:00 2022-11-28T16:38:17.037 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-8 - PowerTools 20 MB/s | 5.3 MB 00:00 2022-11-28T16:38:20.412 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2022-11-28T16:38:21.453 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2022-11-28T16:38:24.651 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 629 kB/s | 24 kB 00:00 2022-11-28T16:38:27.576 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2022-11-28T16:38:27.577 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2022-11-28T16:38:27.704 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-11-28T16:38:27.706 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2022-11-28T16:38:27.706 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-11-28T16:38:27.828 DEBUG:teuthology.orchestra.run.smithi161:> echo no | sudo yum reinstall kernel || true 2022-11-28T16:38:28.430 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:04 ago on Mon 28 Nov 2022 04:38:24 PM UTC. 2022-11-28T16:38:29.530 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-11-28T16:38:29.531 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-11-28T16:38:29.531 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2022-11-28T16:38:29.532 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-11-28T16:38:29.532 INFO:teuthology.orchestra.run.smithi161.stdout:Reinstalling: 2022-11-28T16:38:29.532 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-11-28T16:38:29.533 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-11-28T16:38:29.533 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-11-28T16:38:29.533 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-11-28T16:38:29.534 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-11-28T16:38:29.534 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 8.5 M 2022-11-28T16:38:29.534 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 0 2022-11-28T16:38:29.535 INFO:teuthology.orchestra.run.smithi161.stderr:Operation aborted. 2022-11-28T16:38:29.597 INFO:teuthology.orchestra.run.smithi161.stdout:Is this ok [y/N]: 2022-11-28T16:38:29.597 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum reinstall -y kernel || true 2022-11-28T16:38:30.197 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:06 ago on Mon 28 Nov 2022 04:38:24 PM UTC. 2022-11-28T16:38:31.283 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-11-28T16:38:31.285 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-11-28T16:38:31.285 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2022-11-28T16:38:31.285 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-11-28T16:38:31.286 INFO:teuthology.orchestra.run.smithi161.stdout:Reinstalling: 2022-11-28T16:38:31.286 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 4.18.0-408.el8 CentOS-BaseOS 8.5 M 2022-11-28T16:38:31.287 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-11-28T16:38:31.287 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-11-28T16:38:31.288 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-11-28T16:38:31.288 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-11-28T16:38:31.289 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 8.5 M 2022-11-28T16:38:31.289 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 0 2022-11-28T16:38:31.289 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-11-28T16:38:31.496 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-408.el8.x86_64.rpm 40 MB/s | 8.5 MB 00:00 2022-11-28T16:38:31.496 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-11-28T16:38:31.497 INFO:teuthology.orchestra.run.smithi161.stdout:Total 40 MB/s | 8.5 MB 00:00 2022-11-28T16:38:31.588 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-11-28T16:38:32.027 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-11-28T16:38:32.028 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-11-28T16:38:32.198 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-11-28T16:38:32.232 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-11-28T16:38:32.514 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-11-28T16:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout: Reinstalling : kernel-4.18.0-408.el8.x86_64 1/2 2022-11-28T16:38:32.717 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : kernel-4.18.0-408.el8.x86_64 2/2 2022-11-28T16:38:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-4.18.0-408.el8.x86_64 2/2 2022-11-28T16:38:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 1/2 2022-11-28T16:38:33.587 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-4.18.0-408.el8.x86_64 2/2 2022-11-28T16:38:33.587 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-11-28T16:38:33.588 INFO:teuthology.orchestra.run.smithi161.stdout:Reinstalled: 2022-11-28T16:38:33.588 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-4.18.0-408.el8.x86_64 2022-11-28T16:38:33.588 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-11-28T16:38:33.588 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-11-28T16:38:33.709 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel --last | head -n 1 2022-11-28T16:38:33.837 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-4.18.0-408.el8.x86_64 Mon 28 Nov 2022 04:38:32 PM UTC 2022-11-28T16:38:33.840 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-408.el8.x86_64 2022-11-28T16:38:33.841 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-11-28T16:38:33.841 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-11-28T16:38:33.841 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-11-28T16:38:33.881 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2022-11-28T16:38:33.890 DEBUG:teuthology.parallel:result is None 2022-11-28T16:38:33.891 INFO:teuthology.run_tasks:Running task internal.base... 2022-11-28T16:38:33.899 INFO:teuthology.task.internal:Creating test directory... 2022-11-28T16:38:33.900 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-11-28T16:38:33.953 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-11-28T16:38:33.960 INFO:teuthology.run_tasks:Running task internal.archive... 2022-11-28T16:38:33.973 INFO:teuthology.task.internal:Creating archive directory... 2022-11-28T16:38:33.973 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-11-28T16:38:34.037 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-11-28T16:38:34.049 INFO:teuthology.task.internal:Enabling coredump saving... 2022-11-28T16:38:34.051 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-11-28T16:38:34.100 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-28T16:38:34.109 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-28T16:38:34.111 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-11-28T16:38:34.119 INFO:teuthology.task.internal:Configuring sudo... 2022-11-28T16:38:34.120 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-11-28T16:38:34.179 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-11-28T16:38:34.190 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-11-28T16:38:34.191 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-11-28T16:38:34.236 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-28T16:38:34.323 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-28T16:38:34.371 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-28T16:38:34.466 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-28T16:38:34.514 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-11-28T16:38:34.515 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-11-28T16:38:34.581 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2022-11-28T16:38:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-11-28T16:38:34.964 INFO:teuthology.run_tasks:Running task internal.timer... 2022-11-28T16:38:34.977 INFO:teuthology.task.internal:Starting timer... 2022-11-28T16:38:34.977 INFO:teuthology.run_tasks:Running task pcp... 2022-11-28T16:38:35.002 INFO:teuthology.run_tasks:Running task selinux... 2022-11-28T16:38:35.014 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-11-28T16:38:35.015 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2022-11-28T16:38:35.147 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2022-11-28T16:38:35.148 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-11-28T16:38:35.149 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2022-11-28T16:38:35.184 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2022-11-28T16:38:35.185 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2022-11-28T16:38:35.185 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-11-28T16:38:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T16:38:35.254 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-11-28T16:38:35.254 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2022-11-28T16:38:35.334 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-11-28T16:38:35.349 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-11-28T16:38:35.350 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2022-11-28T16:38:35.377 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-11-28T16:38:35.378 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2022-11-28T16:46:33.485 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734/ansible_failures.yaml 2022-11-28T16:46:33.489 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 422, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 265, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 291, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 316, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi161.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:02:11.325347', 'end': '2022-11-28 16:46:32.902865', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-11-28 16:44:21.577518', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', ',', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}} 2022-11-28T16:46:33.681 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c2740b06feae4b359d6011c117a7ba56 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 422, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 265, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 291, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/task/ansible.py", line 316, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi161.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:02:11.325347', 'end': '2022-11-28 16:46:32.902865', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-11-28 16:44:21.577518', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', ',', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}} 2022-11-28T16:46:33.684 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-11-28T16:46:33.693 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-11-28T16:46:33.693 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-11-28T16:46:33.701 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2022-11-28T16:46:33.788 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-11-28T16:46:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T16:46:33.822 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-11-28T16:46:33.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-11-28T16:46:33.846 INFO:teuthology.task.internal:Duration was 478.868431 seconds 2022-11-28T16:46:33.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-11-28T16:46:33.858 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-11-28T16:46:33.859 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-11-28T16:46:33.912 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-11-28T16:46:34.216 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-11-28T16:46:34.216 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2022-11-28T16:46:34.217 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-11-28T16:46:34.240 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-11-28T16:46:34.241 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-11-28T16:46:34.319 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-11-28T16:46:34.321 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-11-28T16:46:34.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-11-28T16:46:34.429 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-11-28T16:46:34.429 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-11-28T16:46:34.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-11-28T16:46:34.467 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-11-28T16:46:34.524 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2022-11-28T16:46:34.538 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2022-11-28T16:46:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-28T16:46:34.596 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-11-28T16:46:34.605 INFO:teuthology.task.internal:Transferring archived files... 2022-11-28T16:46:34.607 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734/remote/smithi161 2022-11-28T16:46:34.611 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-11-28T16:46:34.698 INFO:teuthology.task.internal:Removing archive directory... 2022-11-28T16:46:34.699 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2022-11-28T16:46:34.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-11-28T16:46:34.730 INFO:teuthology.task.internal:Not uploading archives. 2022-11-28T16:46:34.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-11-28T16:46:34.751 INFO:teuthology.task.internal:Tidying up after the test... 2022-11-28T16:46:34.752 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-11-28T16:46:34.776 INFO:teuthology.orchestra.run.smithi161.stdout: 262206 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 16:46 /home/ubuntu/cephtest 2022-11-28T16:46:34.778 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2022-11-28T16:46:34.805 INFO:teuthology.nuke:Checking targets against current locks 2022-11-28T16:46:34.835 DEBUG:teuthology.nuke:shortname: smithi161 2022-11-28T16:46:34.836 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-11-28T16:46:34.870 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-11-23_21:46:38-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7090734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-11-28 16:33:00.236648', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFjLTrhlQv/MRDlz/xDR+KfvwQGplvud63jKhW4rBQhgjkX1IfQanNVyayVlR8c8VFXyxLPGmBARRIjD9A16Edo='} 2022-11-28T16:46:34.933 INFO:teuthology.orchestra.console:Power off smithi161 2022-11-28T16:46:34.935 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-11-28T16:46:34.962 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-11-28T16:46:38.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-11-28T16:46:43.103 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-11-28T16:46:47.219 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-11-28T16:46:47.433 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2022-11-28T16:46:47.548 INFO:teuthology.run:Summary data: description: rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} duration: 478.86843061447144 failure_reason: '{''smithi161.front.sepia.ceph.com'': {''changed'': True, ''msg'': ''All items completed'', ''results'': [{''_ansible_item_label'': ''ceph/python3-asyncssh'', ''_ansible_no_log'': False, ''ansible_loop_var'': ''item'', ''changed'': True, ''cmd'': [''dnf'', ''-y'', ''copr'', ''enable'', ''ceph/python3-asyncssh''], ''delta'': ''0:02:11.325347'', ''end'': ''2022-11-28 16:46:32.902865'', ''failed'': True, ''invocation'': {''module_args'': {''_raw_params'': ''dnf -y copr enable ceph/python3-asyncssh'', ''_uses_shell'': False, ''argv'': None, ''chdir'': None, ''creates'': None, ''executable'': None, ''removes'': None, ''stdin'': None, ''stdin_add_newline'': True, ''strip_empty_ends'': True, ''warn'': True}}, ''item'': ''ceph/python3-asyncssh'', ''msg'': ''non-zero return code'', ''rc'': 1, ''start'': ''2022-11-28 16:44:21.577518'', ''stderr'': ''Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out'', ''stderr_lines'': [''Enabling a Copr repository. Please note that this repository is not part'', ''of the main distribution, and quality may vary.'', '''', ''The Fedora Project does not exercise any power over the contents of'', ''this repository beyond the rules outlined in the Copr FAQ at'', '','', ''and packages are not held to any quality or security level.'', '''', ''Please do not file bug reports about these packages in Fedora'', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Connection timed out''], ''stdout'': '''', ''stdout_lines'': []}], ''warnings'': ["Consider using the dnf module rather than running ''dnf''. If you need to use command because dnf is insufficient you can add ''warn: false'' to this command task or set ''command_warnings=False'' in ansible.cfg to get rid of this message."]}}' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c2740b06feae4b359d6011c117a7ba56 status: dead success: false 2022-11-28T16:46:47.548 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-11-28T16:46:47.618 INFO:teuthology.run:DEAD