2023-01-25T10:46:26.640 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-25T10:46:26.641 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137291 --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} --name yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137291/orig.config.yaml 2023-01-25T10:46:26.687 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T10:46:26.746 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137291 branch: wip-yuri6-testing-2023-01-24-0746 description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} email: yweinstein@radhat.com first_in_suite: false job_id: '7137291' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-01-24-0746 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_pool_default_min_size: 2 osd_pool_default_size: 2 osd_recovery_max_active: 10 osd_recovery_max_single_start: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd max markdown count: 1000 osd max scrubs: 2 osd mclock override recovery settings: true osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2023-01-24-0746 sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri6-testing-2023-01-24-0746 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c targets: smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTYuTo/n+C846Gze/yaHvARXHI539iadzzB7+MPcOBEkSQase84qfMhMp2JMV75DGixGl3+NCNkKjltDitSxSL3yXd6aQFsnpjdBKjVAGm0Nst4U2PuFJBYwgk6K5sgZ/TrNiCby8o2y9UILOBDvm42heaNZ57hJKGd2uihKfr+7EADgGQUL8r7TdZ4Jpv7QiqL8fS/hp3MdUtYJUUvNXdrgke7azIp0xFNe9Tra+RIUprs/XNrFxPghLhPE5d8Ew71IlLfXiPkovAZS80TeArpCrpd5fVzRTpfpSIhncsS4lU/Vx6HTtQDqpBxZ5ZUMDzEntzn60xRGVCpk5FgbX/QzQxS+2aPda7HiYVunglfVPf6cDV0XRbxfNi7xuupblWZv6rRfDzxyfWTrB8GAwvlZ0Dp93zdSDOAL9EFbb3qLZ+bfEPA3fkkJDxXonBdPeauUtlkWR9uMSfNBJ1gMRYacivuyBPAdvWvbWKnvBiBdl+tUokwLaRtdqaJA+E5bE= smithi080.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIHDBU5pLgczGe7/SMByZJ3XQqWu8JfzaM/bFu6xnzpvtzD6uSQUZsXA4ObSmfM0fEX3zuJP9JvD5nYQFm9kWbvjtG/2unY/xHZ1MCMAtdvRjnsn7YnCupDU4b1nTYNx2mFCJgouRa7SsxuhIt31RUDC4zndu+VN+elEJ8UTgX9hOV3qYKlZhVkpBlmW2DwAgUjmt83xvCoYYNdu9hzYza/kOBBg6OuRuw7ny/ZxIY1rWqvlUdOhldcQlCoRdRvMftNMDsWCuPhXoBUIZhykdGML5lhkqHzOKU9P6DphNTWn6f48jHxALfkpDHCmRN8huc+FS35ly4BKlEfzgprUSRme+88RzjojOqb9kdJJelLuGyK7WUX7WlV+yvU4ovpaOv6c/2NXtwiGLeiX86PXWxQ/h38EC/YiAT92vFst+PEQws6vbiRz2rcO3MiAe2NP8ZsiE8LC2JLnrQ20CJHzQ++C2VNVUQjMHaRHeEYxhtweUhZ9Rz9IKysiecRp/GQvc= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_22:27:30 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-25T10:46:26.746 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa; will attempt to use it 2023-01-25T10:46:26.747 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa/tasks 2023-01-25T10:46:26.747 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-25T10:46:26.749 INFO:teuthology.task.internal:Checking packages... 2023-01-25T10:46:26.776 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '44e4095da12a8269eb6cfe163cbb0d00b1a6412c' 2023-01-25T10:46:26.777 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-25T10:46:26.777 INFO:teuthology.packaging:ref: None 2023-01-25T10:46:26.777 INFO:teuthology.packaging:tag: None 2023-01-25T10:46:26.778 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-01-24-0746 2023-01-25T10:46:26.778 INFO:teuthology.packaging:sha1: 44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T10:46:26.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2023-01-24-0746 2023-01-25T10:46:26.924 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1986.g44e4095d 2023-01-25T10:46:26.925 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-25T10:46:26.931 INFO:teuthology.task.internal:no buildpackages task found 2023-01-25T10:46:26.932 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-25T10:46:26.942 INFO:teuthology.task.internal:Saving configuration 2023-01-25T10:46:26.961 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-25T10:46:26.967 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T10:46:26.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 10:40:04.369890', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJBNYiVoYm6MwMi7hJTQOF1o2f1tKSjHIJM2X8xj6NkhIM/wxZ8j5sMTdy4TkZIUC8saR2EXaMCwPwrU+aXJIZg='} 2023-01-25T10:46:27.012 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137291', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 10:40:04.368634', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZP4GRcmDn3l/Z2r0PEhYSPWX0Gk4Flcp0Di1F6ow/RlwUpBhv/zR27Pw2+rxS9Xb4i+3WwK50yinMrZL9Bmk8='} 2023-01-25T10:46:27.012 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-25T10:46:27.022 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-25T10:46:27.023 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-25T10:46:27.023 INFO:teuthology.run_tasks:Running task console_log... 2023-01-25T10:46:27.086 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fac15ac35e0>, signals=[15]) 2023-01-25T10:46:27.087 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-25T10:46:27.094 INFO:teuthology.task.internal:Opening connections... 2023-01-25T10:46:27.094 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T10:46:27.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:46:27.166 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2023-01-25T10:46:27.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:46:27.237 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-25T10:46:27.244 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2023-01-25T10:46:27.264 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2023-01-25T10:46:27.264 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2023-01-25T10:46:27.327 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2023-01-25T10:46:27.328 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8" 2023-01-25T10:46:27.328 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2023-01-25T10:46:27.328 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2023-01-25T10:46:27.329 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8" 2023-01-25T10:46:27.329 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2023-01-25T10:46:27.329 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-25T10:46:27.329 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2023-01-25T10:46:27.329 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-25T10:46:27.330 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2023-01-25T10:46:27.330 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-25T10:46:27.330 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-25T10:46:27.330 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-25T10:46:27.331 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2023-01-25T10:46:27.347 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2023-01-25T10:46:27.369 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2023-01-25T10:46:27.370 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2023-01-25T10:46:27.430 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2023-01-25T10:46:27.431 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="8" 2023-01-25T10:46:27.431 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2023-01-25T10:46:27.431 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2023-01-25T10:46:27.431 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="8" 2023-01-25T10:46:27.431 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el8" 2023-01-25T10:46:27.432 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-25T10:46:27.432 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2023-01-25T10:46:27.432 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-25T10:46:27.432 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2023-01-25T10:46:27.432 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-25T10:46:27.433 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-25T10:46:27.433 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-25T10:46:27.433 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2023-01-25T10:46:27.447 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-25T10:46:27.458 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-25T10:46:27.478 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-25T10:46:27.479 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2023-01-25T10:46:27.482 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2023-01-25T10:46:27.502 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-25T10:46:27.508 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-25T10:46:27.509 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2023-01-25T10:46:27.540 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2023-01-25T10:46:27.571 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-25T10:46:27.620 INFO:teuthology.run_tasks:Running task kernel... 2023-01-25T10:46:27.635 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-25T10:46:27.636 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-25T10:46:27.636 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T10:46:27.637 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T10:46:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T10:46:27.655 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-01-25T10:46:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T10:46:27.657 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2023-01-25T10:46:27.714 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-408.el8.x86_64 2023-01-25T10:46:27.714 INFO:teuthology.task.kernel:Running kernel on smithi027: 4.18.0-408.el8.x86_64 2023-01-25T10:46:27.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2023-01-25T10:46:27.718 INFO:teuthology.orchestra.run.smithi080.stdout:4.18.0-408.el8.x86_64 2023-01-25T10:46:27.719 INFO:teuthology.task.kernel:Running kernel on smithi080: 4.18.0-408.el8.x86_64 2023-01-25T10:46:27.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum install -y kernel 2023-01-25T10:46:32.302 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - AppStream 15 MB/s | 27 MB 00:01 2023-01-25T10:46:32.505 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - AppStream 16 MB/s | 27 MB 00:01 2023-01-25T10:46:38.285 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - BaseOS 33 MB/s | 27 MB 00:00 2023-01-25T10:46:38.361 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - BaseOS 38 MB/s | 27 MB 00:00 2023-01-25T10:46:42.569 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - Extras 138 kB/s | 18 kB 00:00 2023-01-25T10:46:42.877 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - Extras 153 kB/s | 18 kB 00:00 2023-01-25T10:46:42.980 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - PowerTools 20 MB/s | 5.6 MB 00:00 2023-01-25T10:46:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-25T10:46:44.212 INFO:teuthology.orchestra.run.smithi080.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-01-25T10:46:44.578 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-01-25T10:46:45.471 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 12 MB/s | 13 MB 00:01 2023-01-25T10:46:46.112 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 13 MB 00:01 2023-01-25T10:46:48.800 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 312 kB/s | 24 kB 00:00 2023-01-25T10:46:49.605 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 420 kB/s | 24 kB 00:00 2023-01-25T10:46:51.977 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-25T10:46:51.977 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-25T10:46:52.145 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:46:52.146 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:46:52.146 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:46:52.146 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:46:52.146 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:46:52.147 INFO:teuthology.orchestra.run.smithi080.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-25T10:46:52.147 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-25T10:46:52.147 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-25T10:46:52.147 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:46:52.147 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:46:52.148 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:46:52.148 INFO:teuthology.orchestra.run.smithi080.stdout:Install 3 Packages 2023-01-25T10:46:52.148 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:46:52.149 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 85 M 2023-01-25T10:46:52.149 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 94 M 2023-01-25T10:46:52.149 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:46:52.464 INFO:teuthology.orchestra.run.smithi080.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 29 MB/s | 9.2 MB 00:00 2023-01-25T10:46:52.852 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-25T10:46:52.852 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-25T10:46:53.025 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:46:53.026 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:46:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:46:53.026 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:46:53.026 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:46:53.026 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-25T10:46:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-25T10:46:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-25T10:46:53.027 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:46:53.027 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:46:53.027 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:46:53.028 INFO:teuthology.orchestra.run.smithi027.stdout:Install 3 Packages 2023-01-25T10:46:53.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:46:53.029 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 85 M 2023-01-25T10:46:53.029 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 94 M 2023-01-25T10:46:53.029 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:46:53.372 INFO:teuthology.orchestra.run.smithi080.stdout:(2/3): kernel-core-4.18.0-448.el8.x86_64.rpm 34 MB/s | 41 MB 00:01 2023-01-25T10:46:53.399 INFO:teuthology.orchestra.run.smithi027.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 25 MB/s | 9.2 MB 00:00 2023-01-25T10:46:53.831 INFO:teuthology.orchestra.run.smithi080.stdout:(3/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 20 MB/s | 34 MB 00:01 2023-01-25T10:46:53.833 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:46:53.834 INFO:teuthology.orchestra.run.smithi080.stdout:Total 50 MB/s | 85 MB 00:01 2023-01-25T10:46:54.238 INFO:teuthology.orchestra.run.smithi027.stdout:(2/3): kernel-core-4.18.0-448.el8.x86_64.rpm 34 MB/s | 41 MB 00:01 2023-01-25T10:46:54.470 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:46:54.680 INFO:teuthology.orchestra.run.smithi027.stdout:(3/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 21 MB/s | 34 MB 00:01 2023-01-25T10:46:54.683 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:46:54.683 INFO:teuthology.orchestra.run.smithi027.stdout:Total 51 MB/s | 85 MB 00:01 2023-01-25T10:46:54.857 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:46:54.858 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:46:55.340 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:46:55.746 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:46:55.746 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:46:56.874 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:46:56.926 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:46:57.783 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:46:57.812 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:46:59.639 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:47:00.272 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T10:47:00.681 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:47:01.330 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T10:47:04.166 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T10:47:04.679 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T10:47:05.222 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T10:47:05.662 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T10:47:07.997 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T10:47:08.362 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:09.220 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T10:47:09.491 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:30.283 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:30.311 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:30.836 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:30.837 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-25T10:47:30.837 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-25T10:47:31.458 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:31.458 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:47:31.458 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:47:31.458 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-25T10:47:31.459 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-25T10:47:31.459 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:47:31.459 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:47:31.583 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel --last | head -n 1 2023-01-25T10:47:31.823 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 10:47:07 AM UTC 2023-01-25T10:47:31.825 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T10:47:31.825 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-25T10:47:31.825 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T10:47:31.826 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T10:47:31.827 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-01-25T10:47:31.827 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T10:47:31.827 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel --last | head -n 1 2023-01-25T10:47:32.015 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 10:47:07 AM UTC 2023-01-25T10:47:32.016 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T10:47:32.017 DEBUG:teuthology.orchestra.run.smithi080:> sudo rpm -qi grub2-tools 2023-01-25T10:47:32.050 INFO:teuthology.orchestra.run.smithi080.stdout:Name : grub2-tools 2023-01-25T10:47:32.050 INFO:teuthology.orchestra.run.smithi080.stdout:Epoch : 1 2023-01-25T10:47:32.050 INFO:teuthology.orchestra.run.smithi080.stdout:Version : 2.02 2023-01-25T10:47:32.051 INFO:teuthology.orchestra.run.smithi080.stdout:Release : 129.el8 2023-01-25T10:47:32.051 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: x86_64 2023-01-25T10:47:32.051 INFO:teuthology.orchestra.run.smithi080.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-25T10:47:32.051 INFO:teuthology.orchestra.run.smithi080.stdout:Group : System Environment/Base 2023-01-25T10:47:32.051 INFO:teuthology.orchestra.run.smithi080.stdout:Size : 9265586 2023-01-25T10:47:32.052 INFO:teuthology.orchestra.run.smithi080.stdout:License : GPLv3+ 2023-01-25T10:47:32.052 INFO:teuthology.orchestra.run.smithi080.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-25T10:47:32.052 INFO:teuthology.orchestra.run.smithi080.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-25T10:47:32.052 INFO:teuthology.orchestra.run.smithi080.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-25T10:47:32.053 INFO:teuthology.orchestra.run.smithi080.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-25T10:47:32.053 INFO:teuthology.orchestra.run.smithi080.stdout:Relocations : (not relocatable) 2023-01-25T10:47:32.053 INFO:teuthology.orchestra.run.smithi080.stdout:Packager : CentOS BuildSystem 2023-01-25T10:47:32.053 INFO:teuthology.orchestra.run.smithi080.stdout:Vendor : CentOS 2023-01-25T10:47:32.053 INFO:teuthology.orchestra.run.smithi080.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-25T10:47:32.054 INFO:teuthology.orchestra.run.smithi080.stdout:Summary : Support tools for GRUB. 2023-01-25T10:47:32.054 INFO:teuthology.orchestra.run.smithi080.stdout:Description : 2023-01-25T10:47:32.054 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:47:32.054 INFO:teuthology.orchestra.run.smithi080.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-25T10:47:32.055 INFO:teuthology.orchestra.run.smithi080.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-25T10:47:32.055 INFO:teuthology.orchestra.run.smithi080.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-25T10:47:32.055 INFO:teuthology.orchestra.run.smithi080.stdout:hardware devices. 2023-01-25T10:47:32.056 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:47:32.056 INFO:teuthology.orchestra.run.smithi080.stdout:This subpackage provides tools for support of all platforms. 2023-01-25T10:47:32.057 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-25T10:47:32.057 INFO:teuthology.task.kernel:Updating grub on smithi080 to boot 4.18.0-448.el8.x86_64 2023-01-25T10:47:32.057 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-25T10:47:33.007 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:33.033 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:33.463 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:33.463 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-25T10:47:33.463 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-25T10:47:33.477 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2023-01-25T10:47:34.119 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T10:47:34.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:47:34.120 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:47:34.120 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-25T10:47:34.121 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-25T10:47:34.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:47:34.121 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:47:34.247 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel --last | head -n 1 2023-01-25T10:47:34.454 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 10:47:09 AM UTC 2023-01-25T10:47:34.456 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T10:47:34.457 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-25T10:47:34.457 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T10:47:34.457 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T10:47:34.457 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-25T10:47:34.458 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T10:47:34.458 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel --last | head -n 1 2023-01-25T10:47:34.653 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 10:47:09 AM UTC 2023-01-25T10:47:34.655 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T10:47:34.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2023-01-25T10:47:34.689 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2023-01-25T10:47:34.690 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2023-01-25T10:47:34.690 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.02 2023-01-25T10:47:34.690 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 129.el8 2023-01-25T10:47:34.690 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2023-01-25T10:47:34.690 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-25T10:47:34.691 INFO:teuthology.orchestra.run.smithi027.stdout:Group : System Environment/Base 2023-01-25T10:47:34.691 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 9265586 2023-01-25T10:47:34.691 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2023-01-25T10:47:34.691 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-25T10:47:34.691 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-25T10:47:34.692 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-25T10:47:34.692 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-25T10:47:34.693 INFO:teuthology.orchestra.run.smithi027.stdout:Relocations : (not relocatable) 2023-01-25T10:47:34.693 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : CentOS BuildSystem 2023-01-25T10:47:34.693 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2023-01-25T10:47:34.694 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-25T10:47:34.694 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2023-01-25T10:47:34.694 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2023-01-25T10:47:34.694 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:47:34.694 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-25T10:47:34.695 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-25T10:47:34.695 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-25T10:47:34.695 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2023-01-25T10:47:34.695 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:47:34.695 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2023-01-25T10:47:34.696 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-25T10:47:34.697 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 4.18.0-448.el8.x86_64 2023-01-25T10:47:34.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-25T10:47:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2023-01-25T10:47:36.070 INFO:teuthology.orchestra.run.smithi080.stderr:done 2023-01-25T10:47:36.071 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2023-01-25T10:47:36.117 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.kw9ebxKNSq 2023-01-25T10:47:36.118 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kw9ebxKNSq 2023-01-25T10:47:36.151 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.kw9ebxKNSq 2023-01-25T10:47:36.306 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.kw9ebxKNSq is 6KB 2023-01-25T10:47:36.354 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.kw9ebxKNSq 2023-01-25T10:47:36.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo /bin/ls /boot/loader/entries || true 2023-01-25T10:47:36.440 INFO:teuthology.orchestra.run.smithi080.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-25T10:47:36.441 INFO:teuthology.orchestra.run.smithi080.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-25T10:47:36.441 INFO:teuthology.orchestra.run.smithi080.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-25T10:47:36.441 INFO:teuthology.orchestra.run.smithi080.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-25T10:47:36.442 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-25T10:47:36.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2023-01-25T10:47:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:done 2023-01-25T10:47:37.969 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-01-25T10:47:37.996 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.xzr6LjveM9 2023-01-25T10:47:37.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xzr6LjveM9 2023-01-25T10:47:38.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.xzr6LjveM9 2023-01-25T10:47:38.187 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.xzr6LjveM9 is 6KB 2023-01-25T10:47:38.235 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.xzr6LjveM9 2023-01-25T10:47:38.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2023-01-25T10:47:38.329 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-25T10:47:38.330 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-25T10:47:38.330 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-25T10:47:38.330 INFO:teuthology.orchestra.run.smithi027.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-25T10:47:38.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-25T10:47:38.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2023-01-25T10:48:06.644 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-01-25T10:48:06.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-25T10:48:06.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:48:08.640 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-25T10:48:08.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-25T10:48:08.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:48:25.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2023-01-25T10:48:26.984 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-01-25T10:48:55.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-25T10:48:55.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:48:56.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-25T10:48:56.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:49:00.040 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-01-25T10:49:11.459 DEBUG:teuthology.orchestra.run.smithi080:> true 2023-01-25T10:49:11.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-25T10:49:11.810 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-448.el8.x86_64"... 2023-01-25T10:49:11.810 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2023-01-25T10:49:11.830 INFO:teuthology.orchestra.run.smithi080.stdout:4.18.0-448.el8.x86_64 2023-01-25T10:49:11.831 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-25T10:49:11.831 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T10:49:11.831 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-25T10:49:12.832 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-01-25T10:49:12.833 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T10:49:12.936 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2023-01-25T10:49:12.963 DEBUG:teuthology.parallel:result is None 2023-01-25T10:49:30.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-25T10:49:30.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:49:48.393 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-01-25T10:50:18.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-25T10:50:18.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:50:18.761 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-01-25T10:50:19.064 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-25T10:50:19.064 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-25T10:50:19.064 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-01-25T10:50:19.085 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-448.el8.x86_64 2023-01-25T10:50:19.085 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-25T10:50:19.085 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T10:50:19.086 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-25T10:50:20.086 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-25T10:50:20.087 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T10:50:20.192 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2023-01-25T10:50:20.207 DEBUG:teuthology.parallel:result is None 2023-01-25T10:50:20.207 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-25T10:50:20.214 INFO:teuthology.task.internal:Creating test directory... 2023-01-25T10:50:20.215 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T10:50:20.217 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T10:50:20.242 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-25T10:50:20.249 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-25T10:50:20.257 INFO:teuthology.task.internal:Creating archive directory... 2023-01-25T10:50:20.257 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T10:50:20.278 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T10:50:20.319 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-25T10:50:20.326 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-25T10:50:20.326 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T10:50:20.345 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T10:50:20.384 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:50:20.392 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:50:20.396 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:50:20.404 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T10:50:20.405 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-25T10:50:20.413 INFO:teuthology.task.internal:Configuring sudo... 2023-01-25T10:50:20.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T10:50:20.440 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T10:50:20.475 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-25T10:50:20.489 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-25T10:50:20.490 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T10:50:20.510 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T10:50:20.534 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T10:50:20.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T10:50:20.659 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T10:50:20.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T10:50:20.786 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T10:50:20.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T10:50:20.854 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T10:50:20.896 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T10:50:20.953 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T10:50:21.035 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T10:50:21.082 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T10:50:21.082 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T10:50:21.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2023-01-25T10:50:21.156 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2023-01-25T10:50:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-25T10:50:21.251 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-25T10:50:21.579 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-25T10:50:21.588 INFO:teuthology.task.internal:Starting timer... 2023-01-25T10:50:21.588 INFO:teuthology.run_tasks:Running task pcp... 2023-01-25T10:50:21.598 INFO:teuthology.run_tasks:Running task selinux... 2023-01-25T10:50:21.609 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-25T10:50:21.610 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2023-01-25T10:50:21.670 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2023-01-25T10:50:21.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2023-01-25T10:50:21.746 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2023-01-25T10:50:21.747 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-25T10:50:21.748 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2023-01-25T10:50:21.773 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2023-01-25T10:50:21.774 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2023-01-25T10:50:21.801 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2023-01-25T10:50:21.801 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi080.front.sepia.ceph.com': 'permissive'} 2023-01-25T10:50:21.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-25T10:50:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T10:50:21.842 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-25T10:50:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T10:50:21.870 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-25T10:50:21.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2023-01-25T10:50:21.927 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2023-01-25T10:50:21.955 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-25T10:50:21.970 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-25T10:50:21.971 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-25T10:50:21.991 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-25T10:50:21.992 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi080.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-25T10:56:54.211 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi080.front.sepia.ceph.com')] 2023-01-25T10:56:54.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-01-25T10:56:54.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:56:54.283 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-01-25T10:56:54.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-25T10:56:54.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T10:56:54.419 DEBUG:teuthology.orchestra.run.smithi080:> true 2023-01-25T10:56:54.487 INFO:teuthology.run_tasks:Running task clock... 2023-01-25T10:56:54.499 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-25T10:56:54.500 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-25T10:56:54.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T10:56:54.503 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-25T10:56:54.504 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T10:56:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-25T10:56:54.550 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-25T10:56:54.559 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-25T10:56:54.572 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-25T10:56:54.626 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2023-01-25T10:56:54.632 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2023-01-25T10:56:54.638 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2023-01-25T10:56:54.643 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2023-01-25T10:56:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-25T10:56:54.658 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-25T10:56:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-25T10:56:54.671 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-25T10:56:54.731 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2023-01-25T10:56:54.735 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-25T10:56:54.735 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2023-01-25T10:56:54.735 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.736 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.736 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.736 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.738 INFO:teuthology.orchestra.run.smithi080.stderr:bash: ntpq: command not found 2023-01-25T10:56:54.741 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-25T10:56:54.741 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2023-01-25T10:56:54.741 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.741 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.742 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.742 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T10:56:54.743 INFO:teuthology.run_tasks:Running task install... 2023-01-25T10:56:54.754 DEBUG:teuthology.task.install:project ceph 2023-01-25T10:56:54.755 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'}} 2023-01-25T10:56:54.755 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'} 2023-01-25T10:56:54.755 INFO:teuthology.task.install:Using flavor: default 2023-01-25T10:56:54.761 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-25T10:56:54.762 INFO:teuthology.task.install:extra packages: [] 2023-01-25T10:56:54.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo lsb_release -is 2023-01-25T10:56:54.816 INFO:teuthology.orchestra.run.smithi027.stdout:CentOSStream 2023-01-25T10:56:54.817 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T10:56:54.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo lsb_release -is 2023-01-25T10:56:54.818 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c', 'tag': None, 'wait_for_package': False} 2023-01-25T10:56:54.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo lsb_release -is 2023-01-25T10:56:54.855 INFO:teuthology.orchestra.run.smithi080.stdout:CentOSStream 2023-01-25T10:56:54.856 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T10:56:54.856 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c', 'tag': None, 'wait_for_package': False} 2023-01-25T10:56:54.857 DEBUG:teuthology.orchestra.run.smithi080:> sudo lsb_release -is 2023-01-25T10:56:54.895 INFO:teuthology.orchestra.run.smithi027.stdout:CentOSStream 2023-01-25T10:56:54.896 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T10:56:54.897 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T10:56:54.938 INFO:teuthology.orchestra.run.smithi080.stdout:CentOSStream 2023-01-25T10:56:54.939 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T10:56:54.939 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T10:56:55.059 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/ 2023-01-25T10:56:55.060 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1986.g44e4095d 2023-01-25T10:56:55.089 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/ 2023-01-25T10:56:55.090 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1986.g44e4095d 2023-01-25T10:56:55.318 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-25T10:56:55.318 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T10:56:55.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-25T10:56:55.340 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-01-24-0746/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-25T10:56:55.340 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T10:56:55.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-25T10:56:55.348 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-25T10:56:55.349 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-25T10:56:55.368 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-25T10:56:55.368 DEBUG:teuthology.orchestra.run.smithi080:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/44e4095da12a8269eb6cfe163cbb0d00b1a6412c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-25T10:56:55.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-25T10:56:55.436 DEBUG:teuthology.orchestra.run.smithi080:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-25T10:56:55.498 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-25T10:56:55.513 DEBUG:teuthology.orchestra.run.smithi080:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-25T10:56:55.566 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2023-01-25T10:56:55.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2023-01-25T10:56:55.580 INFO:teuthology.orchestra.run.smithi080.stdout:check_obsoletes = 1 2023-01-25T10:56:55.581 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum clean all 2023-01-25T10:56:55.838 INFO:teuthology.orchestra.run.smithi080.stdout:61 files removed 2023-01-25T10:56:55.839 INFO:teuthology.orchestra.run.smithi027.stdout:61 files removed 2023-01-25T10:56:55.866 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-radosgw 2023-01-25T10:56:55.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2023-01-25T10:56:56.621 INFO:teuthology.orchestra.run.smithi080.stdout:ceph packages for x86_64 226 kB/s | 79 kB 00:00 2023-01-25T10:56:56.630 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 239 kB/s | 79 kB 00:00 2023-01-25T10:56:56.930 INFO:teuthology.orchestra.run.smithi080.stdout:ceph noarch packages 35 kB/s | 9.3 kB 00:00 2023-01-25T10:56:56.940 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 35 kB/s | 9.3 kB 00:00 2023-01-25T10:56:57.214 INFO:teuthology.orchestra.run.smithi080.stdout:ceph source packages 5.8 kB/s | 1.5 kB 00:00 2023-01-25T10:56:57.215 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2023-01-25T10:56:57.991 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - AppStream 37 MB/s | 27 MB 00:00 2023-01-25T10:56:58.118 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - AppStream 31 MB/s | 27 MB 00:00 2023-01-25T10:57:03.684 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - BaseOS 39 MB/s | 27 MB 00:00 2023-01-25T10:57:04.043 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - BaseOS 42 MB/s | 27 MB 00:00 2023-01-25T10:57:07.949 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - Extras 155 kB/s | 18 kB 00:00 2023-01-25T10:57:08.334 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-8 - PowerTools 22 MB/s | 5.6 MB 00:00 2023-01-25T10:57:08.542 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - Extras 156 kB/s | 18 kB 00:00 2023-01-25T10:57:08.944 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-8 - PowerTools 21 MB/s | 5.6 MB 00:00 2023-01-25T10:57:09.591 INFO:teuthology.orchestra.run.smithi080.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-01-25T10:57:10.385 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2023-01-25T10:57:11.417 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 13 MB 00:01 2023-01-25T10:57:12.486 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 13 MB 00:01 2023-01-25T10:57:14.700 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 402 kB/s | 24 kB 00:00 2023-01-25T10:57:15.936 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 467 kB/s | 24 kB 00:00 2023-01-25T10:57:18.138 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:57:18.141 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:57:18.141 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:57:18.141 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:57:18.141 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:57:18.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 13 M 2023-01-25T10:57:18.142 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrading: 2023-01-25T10:57:18.142 INFO:teuthology.orchestra.run.smithi080.stdout: librados2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.6 M 2023-01-25T10:57:18.142 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.6 M 2023-01-25T10:57:18.143 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:57:18.143 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 4.8 M 2023-01-25T10:57:18.143 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 21 M 2023-01-25T10:57:18.143 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 24 k 2023-01-25T10:57:18.143 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-25T10:57:18.144 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 767 k 2023-01-25T10:57:18.144 INFO:teuthology.orchestra.run.smithi080.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-25T10:57:18.144 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-25T10:57:18.144 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 474 k 2023-01-25T10:57:18.144 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-25T10:57:18.145 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 6.5 M 2023-01-25T10:57:18.145 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-25T10:57:18.145 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-25T10:57:18.145 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse 2023-01-25T10:57:18.145 INFO:teuthology.orchestra.run.smithi080.stdout: x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 45 k 2023-01-25T10:57:18.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 115 k 2023-01-25T10:57:18.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 190 k 2023-01-25T10:57:18.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-25T10:57:18.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 372 k 2023-01-25T10:57:18.146 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 364 k 2023-01-25T10:57:18.147 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 106 k 2023-01-25T10:57:18.147 INFO:teuthology.orchestra.run.smithi080.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-25T10:57:18.147 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:18.147 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:57:18.147 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:57:18.148 INFO:teuthology.orchestra.run.smithi080.stdout:Install 21 Packages 2023-01-25T10:57:18.148 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrade 2 Packages 2023-01-25T10:57:18.148 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:18.149 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 58 M 2023-01-25T10:57:18.149 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:57:18.976 INFO:teuthology.orchestra.run.smithi080.stdout:(1/23): ceph-base-18.0.0-1986.g44e4095d.el8.x86 5.8 MB/s | 4.8 MB 00:00 2023-01-25T10:57:19.035 INFO:teuthology.orchestra.run.smithi080.stdout:(2/23): ceph-selinux-18.0.0-1986.g44e4095d.el8. 415 kB/s | 24 kB 00:00 2023-01-25T10:57:19.185 INFO:teuthology.orchestra.run.smithi080.stdout:(3/23): libcephfs2-18.0.0-1986.g44e4095d.el8.x8 5.0 MB/s | 767 kB 00:00 2023-01-25T10:57:19.319 INFO:teuthology.orchestra.run.smithi080.stdout:(4/23): libradosstriper1-18.0.0-1986.g44e4095d. 3.5 MB/s | 474 kB 00:00 2023-01-25T10:57:19.606 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:57:19.609 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:57:19.609 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:57:19.610 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:57:19.610 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:57:19.610 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 13 M 2023-01-25T10:57:19.610 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2023-01-25T10:57:19.610 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.6 M 2023-01-25T10:57:19.611 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.6 M 2023-01-25T10:57:19.611 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:57:19.611 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 4.8 M 2023-01-25T10:57:19.611 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 21 M 2023-01-25T10:57:19.611 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 24 k 2023-01-25T10:57:19.612 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-25T10:57:19.612 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 767 k 2023-01-25T10:57:19.612 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-25T10:57:19.612 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-25T10:57:19.612 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 474 k 2023-01-25T10:57:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-25T10:57:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 6.5 M 2023-01-25T10:57:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-25T10:57:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-25T10:57:19.613 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse 2023-01-25T10:57:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 45 k 2023-01-25T10:57:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 115 k 2023-01-25T10:57:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 190 k 2023-01-25T10:57:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-25T10:57:19.614 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 372 k 2023-01-25T10:57:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 364 k 2023-01-25T10:57:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 106 k 2023-01-25T10:57:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-25T10:57:19.615 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:57:19.615 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:57:19.616 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:57:19.616 INFO:teuthology.orchestra.run.smithi027.stdout:Install 21 Packages 2023-01-25T10:57:19.616 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2023-01-25T10:57:19.616 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:57:19.617 INFO:teuthology.orchestra.run.smithi080.stdout:(5/23): ceph-radosgw-18.0.0-1986.g44e4095d.el8. 8.9 MB/s | 13 MB 00:01 2023-01-25T10:57:19.617 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 58 M 2023-01-25T10:57:19.617 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:57:19.686 INFO:teuthology.orchestra.run.smithi080.stdout:(6/23): python3-ceph-argparse-18.0.0-1986.g44e4 595 kB/s | 45 kB 00:00 2023-01-25T10:57:19.761 INFO:teuthology.orchestra.run.smithi080.stdout:(7/23): python3-ceph-common-18.0.0-1986.g44e409 1.5 MB/s | 115 kB 00:00 2023-01-25T10:57:19.853 INFO:teuthology.orchestra.run.smithi080.stdout:(8/23): python3-cephfs-18.0.0-1986.g44e4095d.el 2.0 MB/s | 190 kB 00:00 2023-01-25T10:57:19.961 INFO:teuthology.orchestra.run.smithi080.stdout:(9/23): python3-rados-18.0.0-1986.g44e4095d.el8 3.4 MB/s | 372 kB 00:00 2023-01-25T10:57:20.054 INFO:teuthology.orchestra.run.smithi080.stdout:(10/23): librgw2-18.0.0-1986.g44e4095d.el8.x86_ 8.8 MB/s | 6.5 MB 00:00 2023-01-25T10:57:20.079 INFO:teuthology.orchestra.run.smithi080.stdout:(11/23): python3-rbd-18.0.0-1986.g44e4095d.el8. 3.0 MB/s | 364 kB 00:00 2023-01-25T10:57:20.137 INFO:teuthology.orchestra.run.smithi080.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2023-01-25T10:57:20.163 INFO:teuthology.orchestra.run.smithi080.stdout:(13/23): python3-rgw-18.0.0-1986.g44e4095d.el8. 979 kB/s | 106 kB 00:00 2023-01-25T10:57:20.197 INFO:teuthology.orchestra.run.smithi080.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-01-25T10:57:20.222 INFO:teuthology.orchestra.run.smithi080.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.0 MB/s | 47 kB 00:00 2023-01-25T10:57:20.247 INFO:teuthology.orchestra.run.smithi080.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 523 kB/s | 44 kB 00:00 2023-01-25T10:57:20.524 INFO:teuthology.orchestra.run.smithi080.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 214 kB/s | 59 kB 00:00 2023-01-25T10:57:20.696 INFO:teuthology.orchestra.run.smithi027.stdout:(1/23): ceph-base-18.0.0-1986.g44e4095d.el8.x86 4.5 MB/s | 4.8 MB 00:01 2023-01-25T10:57:20.746 INFO:teuthology.orchestra.run.smithi027.stdout:(2/23): ceph-selinux-18.0.0-1986.g44e4095d.el8. 487 kB/s | 24 kB 00:00 2023-01-25T10:57:20.748 INFO:teuthology.orchestra.run.smithi080.stdout:(18/23): ceph-common-18.0.0-1986.g44e4095d.el8. 8.3 MB/s | 21 MB 00:02 2023-01-25T10:57:20.773 INFO:teuthology.orchestra.run.smithi080.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 300 kB/s | 75 kB 00:00 2023-01-25T10:57:20.798 INFO:teuthology.orchestra.run.smithi080.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 531 kB/s | 306 kB 00:00 2023-01-25T10:57:20.888 INFO:teuthology.orchestra.run.smithi027.stdout:(3/23): libcephfs2-18.0.0-1986.g44e4095d.el8.x8 5.3 MB/s | 767 kB 00:00 2023-01-25T10:57:21.063 INFO:teuthology.orchestra.run.smithi027.stdout:(4/23): libradosstriper1-18.0.0-1986.g44e4095d. 2.6 MB/s | 474 kB 00:00 2023-01-25T10:57:21.198 INFO:teuthology.orchestra.run.smithi080.stdout:(21/23): librbd1-18.0.0-1986.g44e4095d.el8.x86_ 8.9 MB/s | 3.6 MB 00:00 2023-01-25T10:57:21.257 INFO:teuthology.orchestra.run.smithi080.stdout:(22/23): librados2-18.0.0-1986.g44e4095d.el8.x8 7.5 MB/s | 3.6 MB 00:00 2023-01-25T10:57:21.922 INFO:teuthology.orchestra.run.smithi027.stdout:(5/23): librgw2-18.0.0-1986.g44e4095d.el8.x86_6 7.6 MB/s | 6.5 MB 00:00 2023-01-25T10:57:21.949 INFO:teuthology.orchestra.run.smithi080.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 1.4 MB/s | 1.7 MB 00:01 2023-01-25T10:57:21.952 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:57:21.954 INFO:teuthology.orchestra.run.smithi080.stdout:Total 15 MB/s | 58 MB 00:03 2023-01-25T10:57:21.989 INFO:teuthology.orchestra.run.smithi027.stdout:(6/23): python3-ceph-argparse-18.0.0-1986.g44e4 671 kB/s | 45 kB 00:00 2023-01-25T10:57:21.992 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:57:22.039 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:57:22.039 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:57:22.073 INFO:teuthology.orchestra.run.smithi027.stdout:(7/23): python3-ceph-common-18.0.0-1986.g44e409 1.3 MB/s | 115 kB 00:00 2023-01-25T10:57:22.148 INFO:teuthology.orchestra.run.smithi027.stdout:(8/23): python3-cephfs-18.0.0-1986.g44e4095d.el 2.5 MB/s | 190 kB 00:00 2023-01-25T10:57:22.215 INFO:teuthology.orchestra.run.smithi027.stdout:(9/23): python3-rados-18.0.0-1986.g44e4095d.el8 5.5 MB/s | 372 kB 00:00 2023-01-25T10:57:22.349 INFO:teuthology.orchestra.run.smithi027.stdout:(10/23): ceph-radosgw-18.0.0-1986.g44e4095d.el8 4.8 MB/s | 13 MB 00:02 2023-01-25T10:57:22.374 INFO:teuthology.orchestra.run.smithi027.stdout:(11/23): python3-rbd-18.0.0-1986.g44e4095d.el8. 2.2 MB/s | 364 kB 00:00 2023-01-25T10:57:22.399 INFO:teuthology.orchestra.run.smithi027.stdout:(12/23): python3-rgw-18.0.0-1986.g44e4095d.el8. 2.1 MB/s | 106 kB 00:00 2023-01-25T10:57:22.458 INFO:teuthology.orchestra.run.smithi027.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 4.1 MB/s | 354 kB 00:00 2023-01-25T10:57:22.500 INFO:teuthology.orchestra.run.smithi027.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-01-25T10:57:22.525 INFO:teuthology.orchestra.run.smithi027.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-01-25T10:57:22.550 INFO:teuthology.orchestra.run.smithi027.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2023-01-25T10:57:22.664 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:57:22.668 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:57:22.684 INFO:teuthology.orchestra.run.smithi027.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 441 kB/s | 59 kB 00:00 2023-01-25T10:57:22.768 INFO:teuthology.orchestra.run.smithi027.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 901 kB/s | 75 kB 00:00 2023-01-25T10:57:22.793 INFO:teuthology.orchestra.run.smithi027.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-01-25T10:57:23.168 INFO:teuthology.orchestra.run.smithi027.stdout:(20/23): librados2-18.0.0-1986.g44e4095d.el8.x8 9.7 MB/s | 3.6 MB 00:00 2023-01-25T10:57:23.452 INFO:teuthology.orchestra.run.smithi027.stdout:(21/23): librbd1-18.0.0-1986.g44e4095d.el8.x86_ 13 MB/s | 3.6 MB 00:00 2023-01-25T10:57:23.486 INFO:teuthology.orchestra.run.smithi027.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 2.4 MB/s | 1.7 MB 00:00 2023-01-25T10:57:23.896 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:57:24.131 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-01-25T10:57:24.186 INFO:teuthology.orchestra.run.smithi027.stdout:(23/23): ceph-common-18.0.0-1986.g44e4095d.el8. 4.7 MB/s | 21 MB 00:04 2023-01-25T10:57:24.190 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:57:24.190 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 58 MB 00:04 2023-01-25T10:57:24.230 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:57:24.279 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:57:24.279 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:57:24.615 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-01-25T10:57:24.816 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T10:57:24.925 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:57:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:57:25.225 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T10:57:25.424 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T10:57:25.515 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T10:57:25.780 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 5/25 2023-01-25T10:57:25.949 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 6/25 2023-01-25T10:57:26.163 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:57:26.332 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 6/25 2023-01-25T10:57:26.390 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-01-25T10:57:26.493 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T10:57:26.555 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T10:57:26.731 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-25T10:57:26.834 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-01-25T10:57:26.924 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T10:57:26.990 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T10:57:27.041 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T10:57:27.135 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8. 10/25 2023-01-25T10:57:27.351 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 11/25 2023-01-25T10:57:27.477 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T10:57:27.591 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 12/25 2023-01-25T10:57:27.708 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T10:57:27.741 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 13/25 2023-01-25T10:57:27.812 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 13/25 2023-01-25T10:57:27.821 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T10:57:28.022 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-25T10:57:28.077 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 5/25 2023-01-25T10:57:28.266 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 6/25 2023-01-25T10:57:28.682 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 6/25 2023-01-25T10:57:28.841 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T10:57:28.869 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-25T10:57:28.917 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T10:57:29.049 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 16/25 2023-01-25T10:57:29.099 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-25T10:57:29.109 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 16/25 2023-01-25T10:57:29.233 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T10:57:29.295 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T10:57:29.315 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 17/25 2023-01-25T10:57:29.428 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8. 10/25 2023-01-25T10:57:29.507 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-25T10:57:29.667 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 11/25 2023-01-25T10:57:29.699 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x8 19/25 2023-01-25T10:57:29.887 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 12/25 2023-01-25T10:57:30.041 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 13/25 2023-01-25T10:57:30.109 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 13/25 2023-01-25T10:57:30.346 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-25T10:57:31.174 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-25T10:57:31.349 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 16/25 2023-01-25T10:57:31.422 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 16/25 2023-01-25T10:57:31.616 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 17/25 2023-01-25T10:57:31.801 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-25T10:57:32.000 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x8 19/25 2023-01-25T10:57:33.207 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T10:57:33.456 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T10:57:34.005 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T10:57:34.310 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 21/25 2023-01-25T10:57:34.426 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 21/25 2023-01-25T10:57:34.589 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T10:57:35.713 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T10:57:35.974 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T10:57:36.546 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 20/25 2023-01-25T10:57:36.843 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 21/25 2023-01-25T10:57:36.915 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 21/25 2023-01-25T10:57:37.082 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T10:57:53.754 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T10:57:53.754 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /sys 2023-01-25T10:57:53.755 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /proc 2023-01-25T10:57:53.755 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /dev 2023-01-25T10:57:53.755 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /run 2023-01-25T10:57:53.755 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /mnt 2023-01-25T10:57:53.755 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /var/tmp 2023-01-25T10:57:53.756 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /home 2023-01-25T10:57:53.756 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /tmp 2023-01-25T10:57:53.756 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /dev 2023-01-25T10:57:53.756 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:55.180 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 23/25 2023-01-25T10:57:55.201 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 23/25 2023-01-25T10:57:55.202 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T10:57:55.404 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T10:57:55.404 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T10:57:56.126 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T10:57:56.126 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 1/25 2023-01-25T10:57:56.126 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2/25 2023-01-25T10:57:56.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 3/25 2023-01-25T10:57:56.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T10:57:56.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 5/25 2023-01-25T10:57:56.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 6/25 2023-01-25T10:57:56.127 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T10:57:56.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8. 8/25 2023-01-25T10:57:56.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x8 9/25 2023-01-25T10:57:56.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 10/25 2023-01-25T10:57:56.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 11/25 2023-01-25T10:57:56.128 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 12/25 2023-01-25T10:57:56.129 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 13/25 2023-01-25T10:57:56.129 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-25T10:57:56.129 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-25T10:57:56.129 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-25T10:57:56.129 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-25T10:57:56.130 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-25T10:57:56.130 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-25T10:57:56.130 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-25T10:57:56.130 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-25T10:57:56.130 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T10:57:56.131 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-25T10:57:56.131 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 24/25 2023-01-25T10:57:56.999 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T10:57:56.999 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:56.999 INFO:teuthology.orchestra.run.smithi080.stdout:Upgraded: 2023-01-25T10:57:57.000 INFO:teuthology.orchestra.run.smithi080.stdout: librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.000 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.000 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:57:57.000 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.001 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.001 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.001 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.001 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-25T10:57:57.002 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.002 INFO:teuthology.orchestra.run.smithi080.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-25T10:57:57.002 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-25T10:57:57.002 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.003 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-25T10:57:57.003 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.003 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-25T10:57:57.004 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-25T10:57:57.004 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.004 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.005 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.005 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-25T10:57:57.005 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.006 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.006 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:57:57.007 INFO:teuthology.orchestra.run.smithi080.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-25T10:57:57.007 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:57.007 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:57:57.134 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T10:57:57.134 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2023-01-25T10:57:57.134 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2023-01-25T10:57:57.135 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-01-25T10:57:57.135 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2023-01-25T10:57:57.135 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2023-01-25T10:57:57.135 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2023-01-25T10:57:57.136 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2023-01-25T10:57:57.136 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2023-01-25T10:57:57.136 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-01-25T10:57:57.136 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:57:57.311 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-test 2023-01-25T10:57:58.034 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:44 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:57:58.640 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 23/25 2023-01-25T10:57:58.662 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 23/25 2023-01-25T10:57:58.662 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T10:57:58.865 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T10:57:58.865 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T10:57:59.178 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:57:59.179 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:57:59.179 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:57:59.180 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:57:59.180 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:57:59.180 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 45 M 2023-01-25T10:57:59.180 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:57:59.181 INFO:teuthology.orchestra.run.smithi080.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-25T10:57:59.181 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 137 k 2023-01-25T10:57:59.181 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-25T10:57:59.181 INFO:teuthology.orchestra.run.smithi080.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-25T10:57:59.181 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-25T10:57:59.182 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:59.182 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:57:59.182 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:57:59.182 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2023-01-25T10:57:59.182 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:57:59.183 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 46 M 2023-01-25T10:57:59.183 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 204 M 2023-01-25T10:57:59.183 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:57:59.269 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): jq-1.6-6.el8.x86_64.rpm 2.3 MB/s | 202 kB 00:00 2023-01-25T10:57:59.294 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.6 MB/s | 187 kB 00:00 2023-01-25T10:57:59.328 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 9.3 MB/s | 323 kB 00:00 2023-01-25T10:57:59.353 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-01-25T10:57:59.420 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): libcephsqlite-18.0.0-1986.g44e4095d.el8. 576 kB/s | 137 kB 00:00 2023-01-25T10:57:59.620 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T10:57:59.620 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 1/25 2023-01-25T10:57:59.620 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2/25 2023-01-25T10:57:59.621 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 3/25 2023-01-25T10:57:59.621 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 4/25 2023-01-25T10:57:59.621 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 5/25 2023-01-25T10:57:59.621 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_6 6/25 2023-01-25T10:57:59.621 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 7/25 2023-01-25T10:57:59.622 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8. 8/25 2023-01-25T10:57:59.622 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x8 9/25 2023-01-25T10:57:59.622 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 10/25 2023-01-25T10:57:59.622 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 11/25 2023-01-25T10:57:59.622 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 12/25 2023-01-25T10:57:59.623 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 13/25 2023-01-25T10:57:59.623 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-25T10:57:59.623 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-25T10:57:59.623 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-25T10:57:59.623 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-25T10:57:59.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-25T10:57:59.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-25T10:57:59.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-25T10:57:59.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-25T10:57:59.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 22/25 2023-01-25T10:57:59.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-25T10:57:59.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 24/25 2023-01-25T10:58:00.497 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T10:58:00.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:00.498 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2023-01-25T10:58:00.498 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.498 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.498 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:58:00.498 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.499 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.499 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.499 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.499 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-25T10:58:00.499 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.500 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-25T10:58:00.500 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-25T10:58:00.500 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.500 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-25T10:58:00.500 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.501 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-25T10:58:00.501 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-25T10:58:00.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.501 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-25T10:58:00.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.502 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:00.502 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-25T10:58:00.503 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:00.503 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:00.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2023-01-25T10:58:01.565 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:46 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:02.153 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): ceph-test-18.0.0-1986.g44e4095d.el8.x86_ 15 MB/s | 45 MB 00:02 2023-01-25T10:58:02.156 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:02.156 INFO:teuthology.orchestra.run.smithi080.stdout:Total 15 MB/s | 46 MB 00:02 2023-01-25T10:58:02.196 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:02.216 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:02.217 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:02.510 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:02.513 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:02.784 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:02.786 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:02.786 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:58:02.786 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:02.786 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:58:02.786 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 45 M 2023-01-25T10:58:02.787 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:58:02.787 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-25T10:58:02.787 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 137 k 2023-01-25T10:58:02.787 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-25T10:58:02.787 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-25T10:58:02.788 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-25T10:58:02.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:02.788 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:58:02.788 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2023-01-25T10:58:02.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:02.789 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 46 M 2023-01-25T10:58:02.790 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 204 M 2023-01-25T10:58:02.790 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:58:02.852 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): jq-1.6-6.el8.x86_64.rpm 3.1 MB/s | 202 kB 00:00 2023-01-25T10:58:02.877 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.7 MB/s | 187 kB 00:00 2023-01-25T10:58:02.911 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 2023-01-25T10:58:02.937 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 71 kB 00:00 2023-01-25T10:58:02.974 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:03.020 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): libcephsqlite-18.0.0-1986.g44e4095d.el8. 593 kB/s | 137 kB 00:00 2023-01-25T10:58:03.211 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-01-25T10:58:03.402 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-01-25T10:58:03.608 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T10:58:03.690 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T10:58:03.886 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : jq-1.6-6.el8.x86_64 4/6 2023-01-25T10:58:04.041 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 5/6 2023-01-25T10:58:05.045 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): ceph-test-18.0.0-1986.g44e4095d.el8.x86_ 20 MB/s | 45 MB 00:02 2023-01-25T10:58:05.048 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:05.049 INFO:teuthology.orchestra.run.smithi027.stdout:Total 20 MB/s | 46 MB 00:02 2023-01-25T10:58:05.093 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:58:05.116 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:58:05.116 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:58:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:58:05.418 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:58:05.924 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:58:06.151 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-01-25T10:58:06.378 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-01-25T10:58:06.545 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T10:58:06.625 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T10:58:06.799 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-6.el8.x86_64 4/6 2023-01-25T10:58:06.995 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 5/6 2023-01-25T10:58:08.426 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 5/6 2023-01-25T10:58:08.728 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 6/6 2023-01-25T10:58:09.030 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 6/6 2023-01-25T10:58:09.030 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 1/6 2023-01-25T10:58:09.030 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 2/6 2023-01-25T10:58:09.030 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : jq-1.6-6.el8.x86_64 3/6 2023-01-25T10:58:09.031 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-01-25T10:58:09.031 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-01-25T10:58:09.556 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-01-25T10:58:09.556 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:09.557 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:58:09.557 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:09.557 INFO:teuthology.orchestra.run.smithi080.stdout: jq-1.6-6.el8.x86_64 2023-01-25T10:58:09.557 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:09.557 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-25T10:58:09.558 INFO:teuthology.orchestra.run.smithi080.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-25T10:58:09.558 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-25T10:58:09.558 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:09.558 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:09.745 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph 2023-01-25T10:58:10.478 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:56 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:11.621 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 5/6 2023-01-25T10:58:11.622 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:11.624 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================= 2023-01-25T10:58:11.625 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:58:11.625 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================= 2023-01-25T10:58:11.625 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:58:11.625 INFO:teuthology.orchestra.run.smithi080.stdout: ceph x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 6.4 k 2023-01-25T10:58:11.626 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:58:11.626 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 2.3 M 2023-01-25T10:58:11.626 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 1.5 M 2023-01-25T10:58:11.626 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 236 k 2023-01-25T10:58:11.626 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.9 M 2023-01-25T10:58:11.626 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 17 M 2023-01-25T10:58:11.627 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-25T10:58:11.627 INFO:teuthology.orchestra.run.smithi080.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-25T10:58:11.627 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-25T10:58:11.627 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-25T10:58:11.628 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-25T10:58:11.628 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-25T10:58:11.628 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-25T10:58:11.629 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-25T10:58:11.629 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-25T10:58:11.630 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-25T10:58:11.630 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-25T10:58:11.631 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-25T10:58:11.631 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-25T10:58:11.631 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-25T10:58:11.632 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-25T10:58:11.632 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-25T10:58:11.632 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-25T10:58:11.633 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-25T10:58:11.633 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-25T10:58:11.633 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-25T10:58:11.634 INFO:teuthology.orchestra.run.smithi080.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-25T10:58:11.634 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-25T10:58:11.635 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-25T10:58:11.635 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-25T10:58:11.635 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-25T10:58:11.635 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-25T10:58:11.636 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:11.636 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:58:11.636 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================= 2023-01-25T10:58:11.637 INFO:teuthology.orchestra.run.smithi080.stdout:Install 32 Packages 2023-01-25T10:58:11.637 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:11.638 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 28 M 2023-01-25T10:58:11.638 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 107 M 2023-01-25T10:58:11.639 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:58:11.791 INFO:teuthology.orchestra.run.smithi080.stdout:(1/32): ceph-18.0.0-1986.g44e4095d.el8.x86_64.r 41 kB/s | 6.4 kB 00:00 2023-01-25T10:58:11.883 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 6/6 2023-01-25T10:58:12.021 INFO:teuthology.orchestra.run.smithi080.stdout:(2/32): ceph-mds-18.0.0-1986.g44e4095d.el8.x86_ 5.8 MB/s | 2.3 MB 00:00 2023-01-25T10:58:12.055 INFO:teuthology.orchestra.run.smithi080.stdout:(3/32): ceph-mgr-18.0.0-1986.g44e4095d.el8.x86_ 3.6 MB/s | 1.5 MB 00:00 2023-01-25T10:58:12.174 INFO:teuthology.orchestra.run.smithi080.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1986.g44e4 1.9 MB/s | 236 kB 00:00 2023-01-25T10:58:12.217 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 6/6 2023-01-25T10:58:12.217 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 1/6 2023-01-25T10:58:12.218 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 2/6 2023-01-25T10:58:12.218 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-6.el8.x86_64 3/6 2023-01-25T10:58:12.218 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-01-25T10:58:12.219 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-01-25T10:58:12.239 INFO:teuthology.orchestra.run.smithi080.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 2.4 MB/s | 157 kB 00:00 2023-01-25T10:58:12.264 INFO:teuthology.orchestra.run.smithi080.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2023-01-25T10:58:12.289 INFO:teuthology.orchestra.run.smithi080.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2023-01-25T10:58:12.315 INFO:teuthology.orchestra.run.smithi080.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-25T10:58:12.348 INFO:teuthology.orchestra.run.smithi080.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-25T10:58:12.374 INFO:teuthology.orchestra.run.smithi080.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.3 MB/s | 237 kB 00:00 2023-01-25T10:58:12.408 INFO:teuthology.orchestra.run.smithi080.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2023-01-25T10:58:12.433 INFO:teuthology.orchestra.run.smithi080.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-25T10:58:12.683 INFO:teuthology.orchestra.run.smithi080.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 174 kB/s | 44 kB 00:00 2023-01-25T10:58:12.745 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-01-25T10:58:12.746 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:12.746 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:58:12.746 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:12.746 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-6.el8.x86_64 2023-01-25T10:58:12.747 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:12.747 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-25T10:58:12.747 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-25T10:58:12.747 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-25T10:58:12.747 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:12.748 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:12.792 INFO:teuthology.orchestra.run.smithi080.stdout:(14/32): ceph-mon-18.0.0-1986.g44e4095d.el8.x86 3.9 MB/s | 3.9 MB 00:01 2023-01-25T10:58:12.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2023-01-25T10:58:12.992 INFO:teuthology.orchestra.run.smithi080.stdout:(15/32): ceph-osd-18.0.0-1986.g44e4095d.el8.x86 17 MB/s | 17 MB 00:00 2023-01-25T10:58:13.017 INFO:teuthology.orchestra.run.smithi080.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 554 kB/s | 185 kB 00:00 2023-01-25T10:58:13.068 INFO:teuthology.orchestra.run.smithi080.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-01-25T10:58:13.118 INFO:teuthology.orchestra.run.smithi080.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 352 kB/s | 18 kB 00:00 2023-01-25T10:58:13.177 INFO:teuthology.orchestra.run.smithi080.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 449 kB/s | 173 kB 00:00 2023-01-25T10:58:13.202 INFO:teuthology.orchestra.run.smithi080.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 583 kB/s | 49 kB 00:00 2023-01-25T10:58:13.227 INFO:teuthology.orchestra.run.smithi080.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-01-25T10:58:13.269 INFO:teuthology.orchestra.run.smithi080.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 373 kB/s | 16 kB 00:00 2023-01-25T10:58:13.328 INFO:teuthology.orchestra.run.smithi080.stdout:(23/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2023-01-25T10:58:13.345 INFO:teuthology.orchestra.run.smithi080.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2023-01-25T10:58:13.370 INFO:teuthology.orchestra.run.smithi080.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 569 kB/s | 24 kB 00:00 2023-01-25T10:58:13.395 INFO:teuthology.orchestra.run.smithi080.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-01-25T10:58:13.412 INFO:teuthology.orchestra.run.smithi080.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 435 kB/s | 29 kB 00:00 2023-01-25T10:58:13.430 INFO:teuthology.orchestra.run.smithi080.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 467 kB/s | 27 kB 00:00 2023-01-25T10:58:13.530 INFO:teuthology.orchestra.run.smithi080.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 855 kB/s | 85 kB 00:00 2023-01-25T10:58:13.597 INFO:teuthology.orchestra.run.smithi080.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.2 MB/s | 241 kB 00:00 2023-01-25T10:58:13.622 INFO:teuthology.orchestra.run.smithi080.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2023-01-25T10:58:13.647 INFO:teuthology.orchestra.run.smithi080.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2023-01-25T10:58:13.650 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:13.650 INFO:teuthology.orchestra.run.smithi080.stdout:Total 14 MB/s | 28 MB 00:02 2023-01-25T10:58:13.732 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:58 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:13.733 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:13.771 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:13.771 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:14.150 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:14.152 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:14.957 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:14.961 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2023-01-25T10:58:14.961 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:58:14.961 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2023-01-25T10:58:14.961 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:58:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 6.4 k 2023-01-25T10:58:14.962 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:58:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 2.3 M 2023-01-25T10:58:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 1.5 M 2023-01-25T10:58:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 236 k 2023-01-25T10:58:14.962 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.9 M 2023-01-25T10:58:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 17 M 2023-01-25T10:58:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-25T10:58:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-25T10:58:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-25T10:58:14.963 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-25T10:58:14.964 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-25T10:58:14.964 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-25T10:58:14.964 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-25T10:58:14.964 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-25T10:58:14.964 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-25T10:58:14.965 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-25T10:58:14.965 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-25T10:58:14.965 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-25T10:58:14.965 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-25T10:58:14.965 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-25T10:58:14.966 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-25T10:58:14.966 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-25T10:58:14.966 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-25T10:58:14.966 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-25T10:58:14.966 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-25T10:58:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-25T10:58:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-25T10:58:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-25T10:58:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-25T10:58:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-25T10:58:14.967 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-25T10:58:14.968 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-25T10:58:14.968 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:14.968 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:58:14.968 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2023-01-25T10:58:14.968 INFO:teuthology.orchestra.run.smithi027.stdout:Install 32 Packages 2023-01-25T10:58:14.969 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:14.970 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 28 M 2023-01-25T10:58:14.970 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 107 M 2023-01-25T10:58:14.970 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:58:15.115 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-18.0.0-1986.g44e4095d.el8.x86_64.r 44 kB/s | 6.4 kB 00:00 2023-01-25T10:58:15.407 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-mgr-18.0.0-1986.g44e4095d.el8.x86_ 3.5 MB/s | 1.5 MB 00:00 2023-01-25T10:58:15.457 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): ceph-mds-18.0.0-1986.g44e4095d.el8.x86_ 4.6 MB/s | 2.3 MB 00:00 2023-01-25T10:58:15.481 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:15.541 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1986.g44e4 2.8 MB/s | 236 kB 00:00 2023-01-25T10:58:15.591 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-01-25T10:58:15.608 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 2.3 MB/s | 39 kB 00:00 2023-01-25T10:58:15.625 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 6.1 MB/s | 103 kB 00:00 2023-01-25T10:58:15.650 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-25T10:58:15.684 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 13 MB/s | 457 kB 00:00 2023-01-25T10:58:15.717 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 7.0 MB/s | 237 kB 00:00 2023-01-25T10:58:15.760 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 13 MB/s | 559 kB 00:00 2023-01-25T10:58:15.770 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-25T10:58:15.810 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2023-01-25T10:58:15.943 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): ceph-mon-18.0.0-1986.g44e4095d.el8.x86 4.7 MB/s | 3.9 MB 00:00 2023-01-25T10:58:15.969 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-01-25T10:58:16.005 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-25T10:58:16.111 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.2 MB/s | 173 kB 00:00 2023-01-25T10:58:16.152 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 887 kB/s | 185 kB 00:00 2023-01-25T10:58:16.203 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-01-25T10:58:16.253 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-01-25T10:58:16.276 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-25T10:58:16.303 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 976 kB/s | 49 kB 00:00 2023-01-25T10:58:16.329 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-01-25T10:58:16.354 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-01-25T10:58:16.405 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2023-01-25T10:58:16.472 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 281 kB/s | 19 kB 00:00 2023-01-25T10:58:16.505 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-01-25T10:58:16.530 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2023-01-25T10:58:16.536 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-25T10:58:16.547 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 699 kB/s | 29 kB 00:00 2023-01-25T10:58:16.573 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 655 kB/s | 27 kB 00:00 2023-01-25T10:58:16.689 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.7 MB/s | 241 kB 00:00 2023-01-25T10:58:16.715 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2023-01-25T10:58:16.717 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-25T10:58:16.740 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.6 MB/s | 85 kB 00:00 2023-01-25T10:58:16.766 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2023-01-25T10:58:16.914 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-25T10:58:17.099 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-25T10:58:17.141 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): ceph-osd-18.0.0-1986.g44e4095d.el8.x86 9.5 MB/s | 17 MB 00:01 2023-01-25T10:58:17.143 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:17.144 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 28 MB 00:02 2023-01-25T10:58:17.225 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:58:17.265 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:58:17.265 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:58:17.268 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-25T10:58:17.527 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-25T10:58:17.662 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:58:17.666 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:58:17.790 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-25T10:58:18.045 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-25T10:58:18.278 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-25T10:58:18.682 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-25T10:58:18.911 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:58:18.941 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-25T10:58:19.154 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-25T10:58:19.203 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-25T10:58:19.404 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-25T10:58:19.460 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-25T10:58:19.664 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T10:58:19.728 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-25T10:58:19.947 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-25T10:58:20.000 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-25T10:58:20.160 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-25T10:58:20.180 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-25T10:58:20.370 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-25T10:58:20.398 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-25T10:58:20.563 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-25T10:58:20.609 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-25T10:58:20.741 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-25T10:58:20.894 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-25T10:58:20.976 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-25T10:58:21.119 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-25T10:58:21.229 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-25T10:58:21.365 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-25T10:58:21.451 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-25T10:58:21.689 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-25T10:58:21.717 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-25T10:58:22.076 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-25T10:58:22.097 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-25T10:58:22.355 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-25T10:58:22.418 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8. 27/32 2023-01-25T10:58:22.534 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-25T10:58:22.621 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 28/32 2023-01-25T10:58:22.762 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-25T10:58:23.003 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T10:58:23.316 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-25T10:58:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-25T10:58:23.762 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-25T10:58:23.956 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-25T10:58:24.077 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 28/32 2023-01-25T10:58:24.221 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-25T10:58:24.306 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 29/32 2023-01-25T10:58:24.430 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-25T10:58:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-25T10:58:24.699 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 29/32 2023-01-25T10:58:24.912 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 30/32 2023-01-25T10:58:25.055 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-25T10:58:25.126 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 30/32 2023-01-25T10:58:25.270 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 31/32 2023-01-25T10:58:25.288 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 31/32 2023-01-25T10:58:25.396 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-25T10:58:25.426 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 32/32 2023-01-25T10:58:25.798 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8. 27/32 2023-01-25T10:58:25.862 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 32/32 2023-01-25T10:58:25.971 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 1/32 2023-01-25T10:58:25.972 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 2/32 2023-01-25T10:58:25.972 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 3/32 2023-01-25T10:58:25.973 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 4/32 2023-01-25T10:58:25.973 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 5/32 2023-01-25T10:58:25.973 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8. 6/32 2023-01-25T10:58:25.973 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-25T10:58:25.974 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-25T10:58:25.975 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-25T10:58:25.975 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-25T10:58:25.975 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-25T10:58:25.976 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-25T10:58:25.976 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-25T10:58:25.976 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-25T10:58:25.977 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-25T10:58:25.977 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-25T10:58:25.977 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T10:58:25.978 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-25T10:58:25.978 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-25T10:58:25.980 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-25T10:58:25.980 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-25T10:58:25.980 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-25T10:58:25.981 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-25T10:58:25.981 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-25T10:58:25.981 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-25T10:58:25.981 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-25T10:58:25.981 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-25T10:58:25.982 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-25T10:58:25.982 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-25T10:58:25.982 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-25T10:58:25.982 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-25T10:58:25.983 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 28/32 2023-01-25T10:58:26.852 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-25T10:58:26.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:26.852 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:58:26.853 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:26.853 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:26.853 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:26.853 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:26.853 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:26.854 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:26.854 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-25T10:58:26.854 INFO:teuthology.orchestra.run.smithi080.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-25T10:58:26.854 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-25T10:58:26.854 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-25T10:58:26.855 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-25T10:58:26.855 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-25T10:58:26.855 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-25T10:58:26.855 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-25T10:58:26.855 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-25T10:58:26.856 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-25T10:58:26.856 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-25T10:58:26.856 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-25T10:58:26.856 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-25T10:58:26.856 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend-2.6-1.el8.noarch 2023-01-25T10:58:26.857 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-25T10:58:26.857 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-25T10:58:26.857 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-25T10:58:26.857 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-25T10:58:26.857 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-25T10:58:26.858 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-25T10:58:26.858 INFO:teuthology.orchestra.run.smithi080.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-25T10:58:26.858 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-25T10:58:26.858 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-25T10:58:26.858 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-25T10:58:26.859 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-25T10:58:26.859 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-25T10:58:26.859 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:26.859 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:27.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-base 2023-01-25T10:58:27.511 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 28/32 2023-01-25T10:58:27.728 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 29/32 2023-01-25T10:58:27.814 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:13 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:28.142 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 29/32 2023-01-25T10:58:28.309 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 30/32 2023-01-25T10:58:28.535 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 30/32 2023-01-25T10:58:28.709 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 31/32 2023-01-25T10:58:28.728 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 31/32 2023-01-25T10:58:28.858 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:58:28.859 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 32/32 2023-01-25T10:58:28.956 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:28.957 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:58:28.957 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:29.260 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install cephadm 2023-01-25T10:58:29.329 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 32/32 2023-01-25T10:58:29.329 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 1/32 2023-01-25T10:58:29.330 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 2/32 2023-01-25T10:58:29.330 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 3/32 2023-01-25T10:58:29.330 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 4/32 2023-01-25T10:58:29.330 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 5/32 2023-01-25T10:58:29.330 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8. 6/32 2023-01-25T10:58:29.330 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-25T10:58:29.331 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-25T10:58:29.331 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-25T10:58:29.331 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-25T10:58:29.332 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-25T10:58:29.332 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-25T10:58:29.332 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-25T10:58:29.333 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-25T10:58:29.333 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-25T10:58:29.333 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-25T10:58:29.333 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T10:58:29.334 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-25T10:58:29.334 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-25T10:58:29.334 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-25T10:58:29.334 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-25T10:58:29.335 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-25T10:58:29.335 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-25T10:58:29.335 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-25T10:58:29.335 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-25T10:58:29.336 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-25T10:58:29.337 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-25T10:58:29.337 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-25T10:58:29.337 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-25T10:58:29.337 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-25T10:58:29.337 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-25T10:58:29.976 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:15 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:30.217 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-25T10:58:30.217 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:30.217 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:58:30.218 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:30.218 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:30.218 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:30.219 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:30.219 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:30.219 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:30.219 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-25T10:58:30.219 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-25T10:58:30.220 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-25T10:58:30.220 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-25T10:58:30.220 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-25T10:58:30.220 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-25T10:58:30.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-25T10:58:30.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-25T10:58:30.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-25T10:58:30.222 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-25T10:58:30.222 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-25T10:58:30.222 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-25T10:58:30.222 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-25T10:58:30.223 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2023-01-25T10:58:30.223 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-25T10:58:30.223 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-25T10:58:30.224 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-25T10:58:30.225 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-25T10:58:30.225 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-25T10:58:30.225 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-25T10:58:30.226 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-25T10:58:30.226 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-25T10:58:30.226 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-25T10:58:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-25T10:58:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-25T10:58:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-25T10:58:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:30.228 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:30.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2023-01-25T10:58:31.149 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:31.150 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:31.150 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:58:31.150 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:31.151 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:58:31.151 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 82 k 2023-01-25T10:58:31.151 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:31.151 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:58:31.151 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:31.152 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:58:31.152 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:31.152 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 82 k 2023-01-25T10:58:31.152 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 374 k 2023-01-25T10:58:31.153 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:58:31.185 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:16 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:31.335 INFO:teuthology.orchestra.run.smithi080.stdout:cephadm-18.0.0-1986.g44e4095d.el8.noarch.rpm 449 kB/s | 82 kB 00:00 2023-01-25T10:58:31.335 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:31.336 INFO:teuthology.orchestra.run.smithi080.stdout:Total 446 kB/s | 82 kB 00:00 2023-01-25T10:58:31.336 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:31.343 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:31.343 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:31.345 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:31.348 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:31.406 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:32.303 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:58:32.413 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:32.414 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:58:32.415 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:32.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2023-01-25T10:58:33.007 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:33.162 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:33.397 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:33.605 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:18 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:33.850 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:33.850 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:33.851 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:58:33.851 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:33.851 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:33.851 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:34.111 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-immutable-object-cache 2023-01-25T10:58:34.823 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:34.824 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:34.824 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:58:34.824 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:34.825 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:58:34.825 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 82 k 2023-01-25T10:58:34.825 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:34.825 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:58:34.825 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:34.826 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:58:34.826 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:34.826 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 82 k 2023-01-25T10:58:34.826 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 374 k 2023-01-25T10:58:34.827 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:58:34.842 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:20 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:35.240 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-18.0.0-1986.g44e4095d.el8.noarch.rpm 199 kB/s | 82 kB 00:00 2023-01-25T10:58:35.240 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:35.241 INFO:teuthology.orchestra.run.smithi027.stdout:Total 198 kB/s | 82 kB 00:00 2023-01-25T10:58:35.241 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:58:35.249 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:58:35.249 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:58:35.251 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:58:35.255 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:58:35.300 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:58:35.998 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:35.999 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:35.999 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2023-01-25T10:58:35.999 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:35.999 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:58:35.999 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 165 k 2023-01-25T10:58:36.000 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:36.000 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:58:36.000 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:36.000 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:58:36.000 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:36.001 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 165 k 2023-01-25T10:58:36.001 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 503 k 2023-01-25T10:58:36.001 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:58:36.224 INFO:teuthology.orchestra.run.smithi080.stdout:ceph-immutable-object-cache-18.0.0-1986.g44e409 738 kB/s | 165 kB 00:00 2023-01-25T10:58:36.224 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:36.225 INFO:teuthology.orchestra.run.smithi080.stdout:Total 734 kB/s | 165 kB 00:00 2023-01-25T10:58:36.225 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:36.236 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:36.236 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:36.285 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:36.288 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:36.447 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:36.609 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T10:58:36.928 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:37.034 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T10:58:37.102 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:37.350 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:37.530 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T10:58:37.530 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:37.530 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:58:37.531 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:37.531 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:37.531 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:37.743 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr 2023-01-25T10:58:37.915 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:58:37.916 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:37.916 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:58:37.916 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:37.916 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:37.917 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:38.472 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:24 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:38.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2023-01-25T10:58:39.518 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:58:39.567 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:24 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:39.616 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:39.617 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:58:39.617 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:39.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-dashboard 2023-01-25T10:58:40.411 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:26 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:40.787 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:40.787 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-25T10:58:40.788 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:40.788 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:58:40.788 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 165 k 2023-01-25T10:58:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:40.789 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:58:40.789 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:40.789 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:58:40.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:40.790 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 165 k 2023-01-25T10:58:40.791 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 503 k 2023-01-25T10:58:40.791 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:58:41.037 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-18.0.0-1986.g44e409 667 kB/s | 165 kB 00:00 2023-01-25T10:58:41.037 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:41.038 INFO:teuthology.orchestra.run.smithi027.stdout:Total 664 kB/s | 165 kB 00:00 2023-01-25T10:58:41.038 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:58:41.050 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:58:41.050 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:58:41.104 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:58:41.107 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:58:41.241 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:58:41.415 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T10:58:41.567 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:41.568 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:41.568 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:58:41.568 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:41.568 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:58:41.569 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 1.7 M 2023-01-25T10:58:41.569 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:58:41.569 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 23 k 2023-01-25T10:58:41.569 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 15 k 2023-01-25T10:58:41.569 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-25T10:58:41.570 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-25T10:58:41.570 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-25T10:58:41.570 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:41.570 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:58:41.570 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:58:41.571 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2023-01-25T10:58:41.571 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:41.571 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 2.0 M 2023-01-25T10:58:41.572 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 8.2 M 2023-01-25T10:58:41.572 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:58:41.860 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): ceph-grafana-dashboards-18.0.0-1986.g44e 79 kB/s | 23 kB 00:00 2023-01-25T10:58:41.863 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T10:58:41.885 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): ceph-prometheus-alerts-18.0.0-1986.g44e4 46 kB/s | 15 kB 00:00 2023-01-25T10:58:41.944 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 511 kB/s | 43 kB 00:00 2023-01-25T10:58:42.001 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 291 kB/s | 34 kB 00:00 2023-01-25T10:58:42.068 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): ceph-mgr-dashboard-18.0.0-1986.g44e4095d 3.5 MB/s | 1.7 MB 00:00 2023-01-25T10:58:42.127 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-01-25T10:58:42.129 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:42.129 INFO:teuthology.orchestra.run.smithi080.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2023-01-25T10:58:42.139 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:42.148 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:42.148 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:42.229 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:42.232 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:42.361 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d. 1/1 2023-01-25T10:58:42.361 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:42.362 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:58:42.362 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:58:42.362 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:42.362 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:42.503 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:42.591 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2023-01-25T10:58:42.731 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-25T10:58:42.950 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-25T10:58:43.127 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-25T10:58:43.280 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.n 4/6 2023-01-25T10:58:43.354 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:28 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:43.638 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8. 5/6 2023-01-25T10:58:43.893 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 6/6 2023-01-25T10:58:44.193 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 6/6 2023-01-25T10:58:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8. 1/6 2023-01-25T10:58:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 2/6 2023-01-25T10:58:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.n 3/6 2023-01-25T10:58:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-25T10:58:44.194 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-25T10:58:44.460 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:58:44.570 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:44.571 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:58:44.571 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:44.656 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-25T10:58:44.656 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:44.656 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:58:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-25T10:58:44.657 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-25T10:58:44.658 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-25T10:58:44.658 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:44.658 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:44.769 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-25T10:58:44.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2023-01-25T10:58:45.489 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:31 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:45.651 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:30 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:46.625 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:46.626 INFO:teuthology.orchestra.run.smithi080.stdout:===================================================================================================== 2023-01-25T10:58:46.626 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:58:46.626 INFO:teuthology.orchestra.run.smithi080.stdout:===================================================================================================== 2023-01-25T10:58:46.627 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:58:46.627 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 7.4 M 2023-01-25T10:58:46.627 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:58:46.627 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-25T10:58:46.628 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-25T10:58:46.628 INFO:teuthology.orchestra.run.smithi080.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-25T10:58:46.628 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-25T10:58:46.628 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-25T10:58:46.628 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-25T10:58:46.629 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-25T10:58:46.629 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:46.629 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:58:46.629 INFO:teuthology.orchestra.run.smithi080.stdout:===================================================================================================== 2023-01-25T10:58:46.629 INFO:teuthology.orchestra.run.smithi080.stdout:Install 8 Packages 2023-01-25T10:58:46.630 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:46.630 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 35 M 2023-01-25T10:58:46.630 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 210 M 2023-01-25T10:58:46.631 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:58:46.875 INFO:teuthology.orchestra.run.smithi080.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2023-01-25T10:58:46.886 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:46.887 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:46.887 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:58:46.887 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:46.887 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:58:46.888 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 1.7 M 2023-01-25T10:58:46.888 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:58:46.888 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 23 k 2023-01-25T10:58:46.888 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 15 k 2023-01-25T10:58:46.888 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-25T10:58:46.889 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-25T10:58:46.889 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-25T10:58:46.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:46.889 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:58:46.890 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:58:46.890 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2023-01-25T10:58:46.890 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:46.891 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.0 M 2023-01-25T10:58:46.891 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 8.2 M 2023-01-25T10:58:46.891 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:58:46.983 INFO:teuthology.orchestra.run.smithi080.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 35 MB/s | 3.7 MB 00:00 2023-01-25T10:58:47.050 INFO:teuthology.orchestra.run.smithi080.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2023-01-25T10:58:47.073 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): ceph-prometheus-alerts-18.0.0-1986.g44e4 79 kB/s | 15 kB 00:00 2023-01-25T10:58:47.075 INFO:teuthology.orchestra.run.smithi080.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-01-25T10:58:47.099 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): ceph-grafana-dashboards-18.0.0-1986.g44e 110 kB/s | 23 kB 00:00 2023-01-25T10:58:47.134 INFO:teuthology.orchestra.run.smithi080.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2023-01-25T10:58:47.157 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 509 kB/s | 43 kB 00:00 2023-01-25T10:58:47.167 INFO:teuthology.orchestra.run.smithi080.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-01-25T10:58:47.208 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 309 kB/s | 34 kB 00:00 2023-01-25T10:58:47.270 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): ceph-mgr-dashboard-18.0.0-1986.g44e4095d 4.5 MB/s | 1.7 MB 00:00 2023-01-25T10:58:47.345 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-01-25T10:58:47.347 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:47.347 INFO:teuthology.orchestra.run.smithi027.stdout:Total 4.4 MB/s | 2.0 MB 00:00 2023-01-25T10:58:47.358 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:58:47.368 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:58:47.368 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:58:47.409 INFO:teuthology.orchestra.run.smithi080.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 41 MB/s | 14 MB 00:00 2023-01-25T10:58:47.455 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:58:47.459 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:58:47.610 INFO:teuthology.orchestra.run.smithi080.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-198 7.5 MB/s | 7.4 MB 00:00 2023-01-25T10:58:47.612 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:47.612 INFO:teuthology.orchestra.run.smithi080.stdout:Total 36 MB/s | 35 MB 00:00 2023-01-25T10:58:47.749 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:58:47.843 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:47.861 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:47.861 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:47.964 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-25T10:58:48.167 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-25T10:58:48.259 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:48.263 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:48.401 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-25T10:58:48.529 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.n 4/6 2023-01-25T10:58:48.858 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:48.909 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8. 5/6 2023-01-25T10:58:49.048 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T10:58:49.175 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 6/6 2023-01-25T10:58:49.192 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T10:58:49.339 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T10:58:49.493 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 6/6 2023-01-25T10:58:49.493 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8. 1/6 2023-01-25T10:58:49.493 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarc 2/6 2023-01-25T10:58:49.494 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.n 3/6 2023-01-25T10:58:49.494 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-25T10:58:49.494 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-25T10:58:49.937 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T10:58:50.012 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-25T10:58:50.012 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:50.012 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:58:50.013 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:50.013 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:50.013 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:50.013 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-25T10:58:50.013 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-25T10:58:50.014 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-25T10:58:50.014 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:50.014 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:58:50.114 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T10:58:50.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-25T10:58:50.768 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T10:58:50.892 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:35 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:58:50.939 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T10:58:51.617 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T10:58:52.113 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:58:52.114 INFO:teuthology.orchestra.run.smithi027.stdout:===================================================================================================== 2023-01-25T10:58:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:58:52.115 INFO:teuthology.orchestra.run.smithi027.stdout:===================================================================================================== 2023-01-25T10:58:52.115 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:58:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 7.4 M 2023-01-25T10:58:52.115 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:58:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-25T10:58:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-25T10:58:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-25T10:58:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-25T10:58:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-25T10:58:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-25T10:58:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-25T10:58:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:52.117 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:58:52.118 INFO:teuthology.orchestra.run.smithi027.stdout:===================================================================================================== 2023-01-25T10:58:52.118 INFO:teuthology.orchestra.run.smithi027.stdout:Install 8 Packages 2023-01-25T10:58:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:58:52.119 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T10:58:52.119 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 35 M 2023-01-25T10:58:52.119 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 210 M 2023-01-25T10:58:52.119 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:58:52.292 INFO:teuthology.orchestra.run.smithi027.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-01-25T10:58:52.339 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T10:58:52.401 INFO:teuthology.orchestra.run.smithi027.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 35 MB/s | 3.7 MB 00:00 2023-01-25T10:58:52.434 INFO:teuthology.orchestra.run.smithi027.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-01-25T10:58:52.501 INFO:teuthology.orchestra.run.smithi027.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2023-01-25T10:58:52.560 INFO:teuthology.orchestra.run.smithi027.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2023-01-25T10:58:52.594 INFO:teuthology.orchestra.run.smithi027.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-01-25T10:58:52.861 INFO:teuthology.orchestra.run.smithi027.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 32 MB/s | 14 MB 00:00 2023-01-25T10:58:54.098 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T10:58:55.335 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-25T10:58:55.521 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 8/8 2023-01-25T10:58:55.842 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 8/8 2023-01-25T10:58:55.842 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 1/8 2023-01-25T10:58:55.842 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-25T10:58:55.842 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-25T10:58:55.843 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-25T10:58:55.843 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T10:58:55.843 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-25T10:58:55.843 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-25T10:58:56.373 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-25T10:58:56.373 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:56.373 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:58:56.373 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:58:56.373 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-25T10:58:56.374 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-25T10:58:56.374 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-25T10:58:56.374 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-25T10:58:56.374 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-25T10:58:56.374 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-25T10:58:56.375 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-25T10:58:56.375 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:56.375 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:58:56.485 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-rook 2023-01-25T10:58:56.987 INFO:teuthology.orchestra.run.smithi027.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-198 1.5 MB/s | 7.4 MB 00:04 2023-01-25T10:58:56.990 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:56.990 INFO:teuthology.orchestra.run.smithi027.stdout:Total 7.3 MB/s | 35 MB 00:04 2023-01-25T10:58:57.206 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:43 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:58:57.230 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:58:57.249 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:58:57.249 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:58:57.650 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:58:57.654 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:58:58.227 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:58:58.344 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:58:58.346 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2023-01-25T10:58:58.346 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:58:58.346 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2023-01-25T10:58:58.347 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:58:58.347 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 47 k 2023-01-25T10:58:58.347 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:58:58.347 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-25T10:58:58.348 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-25T10:58:58.348 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-25T10:58:58.348 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-25T10:58:58.348 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-25T10:58:58.348 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-25T10:58:58.349 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-25T10:58:58.349 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-25T10:58:58.349 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-25T10:58:58.349 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-25T10:58:58.350 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-25T10:58:58.350 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-25T10:58:58.350 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:58.350 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:58:58.350 INFO:teuthology.orchestra.run.smithi080.stdout:======================================================================================== 2023-01-25T10:58:58.351 INFO:teuthology.orchestra.run.smithi080.stdout:Install 13 Packages 2023-01-25T10:58:58.351 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:58:58.352 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 1.9 M 2023-01-25T10:58:58.352 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 25 M 2023-01-25T10:58:58.352 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:58:58.388 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T10:58:58.427 INFO:teuthology.orchestra.run.smithi080.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 343 kB/s | 27 kB 00:00 2023-01-25T10:58:58.452 INFO:teuthology.orchestra.run.smithi080.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 195 kB/s | 20 kB 00:00 2023-01-25T10:58:58.478 INFO:teuthology.orchestra.run.smithi080.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-01-25T10:58:58.503 INFO:teuthology.orchestra.run.smithi080.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-25T10:58:58.522 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T10:58:58.528 INFO:teuthology.orchestra.run.smithi080.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2023-01-25T10:58:58.545 INFO:teuthology.orchestra.run.smithi080.stdout:(6/13): ceph-mgr-rook-18.0.0-1986.g44e4095d.el8 242 kB/s | 47 kB 00:00 2023-01-25T10:58:58.562 INFO:teuthology.orchestra.run.smithi080.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 719 kB/s | 43 kB 00:00 2023-01-25T10:58:58.646 INFO:teuthology.orchestra.run.smithi080.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 281 kB/s | 33 kB 00:00 2023-01-25T10:58:58.663 INFO:teuthology.orchestra.run.smithi080.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2023-01-25T10:58:58.671 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T10:58:58.697 INFO:teuthology.orchestra.run.smithi080.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 606 kB/s | 81 kB 00:00 2023-01-25T10:58:58.739 INFO:teuthology.orchestra.run.smithi080.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 813 kB/s | 61 kB 00:00 2023-01-25T10:58:58.756 INFO:teuthology.orchestra.run.smithi080.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-01-25T10:58:59.023 INFO:teuthology.orchestra.run.smithi080.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.1 MB/s | 1.2 MB 00:00 2023-01-25T10:58:59.025 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:58:59.025 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.8 MB/s | 1.9 MB 00:00 2023-01-25T10:58:59.077 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:58:59.092 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:58:59.092 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:58:59.191 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:58:59.194 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:58:59.290 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T10:58:59.471 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T10:58:59.709 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:58:59.980 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-25T10:59:00.127 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T10:59:00.196 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-25T10:59:00.422 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-25T10:59:00.652 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-25T10:59:00.817 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-25T10:59:01.023 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-25T10:59:01.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T10:59:01.325 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-25T10:59:01.582 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-25T10:59:01.816 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T10:59:02.171 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T10:59:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T10:59:02.652 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-25T10:59:02.960 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-25T10:59:03.299 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-25T10:59:04.213 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-25T10:59:04.278 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T10:59:04.336 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 13/13 2023-01-25T10:59:04.654 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 13/13 2023-01-25T10:59:04.654 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 1/13 2023-01-25T10:59:04.655 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-25T10:59:04.655 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-25T10:59:04.655 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-25T10:59:04.655 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-25T10:59:04.655 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-25T10:59:04.656 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-25T10:59:04.656 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-25T10:59:04.656 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-25T10:59:04.656 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-25T10:59:04.656 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-25T10:59:04.657 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-25T10:59:05.242 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-25T10:59:05.243 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:05.243 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:05.243 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:05.243 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-25T10:59:05.243 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-25T10:59:05.244 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-25T10:59:05.244 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-25T10:59:05.244 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-25T10:59:05.244 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-25T10:59:05.244 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-25T10:59:05.245 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-25T10:59:05.245 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-25T10:59:05.245 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-25T10:59:05.245 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-25T10:59:05.245 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-25T10:59:05.246 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:05.246 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:05.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-cephadm 2023-01-25T10:59:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-25T10:59:05.787 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 8/8 2023-01-25T10:59:06.117 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:52 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:06.133 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 8/8 2023-01-25T10:59:06.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095 1/8 2023-01-25T10:59:06.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-25T10:59:06.133 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-25T10:59:06.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-25T10:59:06.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T10:59:06.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-25T10:59:06.134 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-25T10:59:07.139 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-25T10:59:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:07.140 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-25T10:59:07.140 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-25T10:59:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-25T10:59:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-25T10:59:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-25T10:59:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-25T10:59:07.141 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-25T10:59:07.142 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:07.142 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:07.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2023-01-25T10:59:07.260 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:07.261 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:07.261 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:59:07.261 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:07.261 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:07.262 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 120 k 2023-01-25T10:59:07.262 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:59:07.262 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-25T10:59:07.262 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-25T10:59:07.262 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-25T10:59:07.263 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-25T10:59:07.263 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:07.263 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:07.263 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:07.263 INFO:teuthology.orchestra.run.smithi080.stdout:Install 5 Packages 2023-01-25T10:59:07.263 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:07.264 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 6.0 M 2023-01-25T10:59:07.264 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 26 M 2023-01-25T10:59:07.265 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:07.356 INFO:teuthology.orchestra.run.smithi080.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.7 MB/s | 538 kB 00:00 2023-01-25T10:59:07.448 INFO:teuthology.orchestra.run.smithi080.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-01-25T10:59:07.599 INFO:teuthology.orchestra.run.smithi080.stdout:(3/5): ceph-mgr-cephadm-18.0.0-1986.g44e4095d.e 358 kB/s | 120 kB 00:00 2023-01-25T10:59:07.624 INFO:teuthology.orchestra.run.smithi080.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 344 kB/s | 60 kB 00:00 2023-01-25T10:59:07.816 INFO:teuthology.orchestra.run.smithi080.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2023-01-25T10:59:07.817 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:07.818 INFO:teuthology.orchestra.run.smithi080.stdout:Total 11 MB/s | 6.0 MB 00:00 2023-01-25T10:59:07.868 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:07.879 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:07.880 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:08.007 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:53 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:08.022 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:08.026 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:08.344 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:08.641 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-25T10:59:09.253 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2023-01-25T10:59:09.255 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:59:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2023-01-25T10:59:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:09.256 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 47 k 2023-01-25T10:59:09.256 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:59:09.256 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-25T10:59:09.256 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-25T10:59:09.257 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-25T10:59:09.257 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-25T10:59:09.257 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-25T10:59:09.257 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-25T10:59:09.257 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-25T10:59:09.258 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-25T10:59:09.258 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-25T10:59:09.258 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-25T10:59:09.258 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-25T10:59:09.258 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-25T10:59:09.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:09.259 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:09.259 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2023-01-25T10:59:09.259 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2023-01-25T10:59:09.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:09.260 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 1.9 M 2023-01-25T10:59:09.260 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 25 M 2023-01-25T10:59:09.260 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:09.320 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 435 kB/s | 27 kB 00:00 2023-01-25T10:59:09.345 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 232 kB/s | 20 kB 00:00 2023-01-25T10:59:09.371 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-01-25T10:59:09.396 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-25T10:59:09.421 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2023-01-25T10:59:09.446 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): ceph-mgr-rook-18.0.0-1986.g44e4095d.el8 252 kB/s | 47 kB 00:00 2023-01-25T10:59:09.472 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 562 kB/s | 43 kB 00:00 2023-01-25T10:59:09.556 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 150 kB/s | 16 kB 00:00 2023-01-25T10:59:09.581 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 207 kB/s | 33 kB 00:00 2023-01-25T10:59:09.595 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-25T10:59:09.606 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 605 kB/s | 81 kB 00:00 2023-01-25T10:59:09.665 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 734 kB/s | 61 kB 00:00 2023-01-25T10:59:09.690 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 734 kB/s | 61 kB 00:00 2023-01-25T10:59:09.996 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-25T10:59:10.232 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2023-01-25T10:59:10.234 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:10.234 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2023-01-25T10:59:10.271 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-25T10:59:10.294 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:10.310 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:10.311 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:10.423 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:10.468 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 5/5 2023-01-25T10:59:10.779 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 5/5 2023-01-25T10:59:10.780 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/5 2023-01-25T10:59:10.780 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-25T10:59:10.780 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-25T10:59:10.780 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-25T10:59:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:11.216 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-25T10:59:11.269 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-25T10:59:11.270 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:11.270 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:11.270 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:11.270 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-25T10:59:11.270 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-25T10:59:11.271 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-25T10:59:11.271 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-25T10:59:11.271 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:11.271 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:11.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-fuse 2023-01-25T10:59:11.414 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-25T10:59:11.681 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-25T10:59:11.927 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-25T10:59:12.101 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:58 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:12.352 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-25T10:59:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-25T10:59:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-25T10:59:13.151 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-25T10:59:13.247 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:13.247 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:13.247 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:59:13.248 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:13.248 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:13.248 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 845 k 2023-01-25T10:59:13.248 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2023-01-25T10:59:13.248 INFO:teuthology.orchestra.run.smithi080.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-25T10:59:13.249 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:13.249 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:13.249 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:13.249 INFO:teuthology.orchestra.run.smithi080.stdout:Install 2 Packages 2023-01-25T10:59:13.249 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:13.250 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 928 k 2023-01-25T10:59:13.250 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 2.9 M 2023-01-25T10:59:13.250 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:13.344 INFO:teuthology.orchestra.run.smithi080.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 879 kB/s | 83 kB 00:00 2023-01-25T10:59:13.565 INFO:teuthology.orchestra.run.smithi080.stdout:(2/2): ceph-fuse-18.0.0-1986.g44e4095d.el8.x86_ 2.6 MB/s | 845 kB 00:00 2023-01-25T10:59:13.566 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:13.567 INFO:teuthology.orchestra.run.smithi080.stdout:Total 2.8 MB/s | 928 kB 00:00 2023-01-25T10:59:13.577 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:13.597 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:13.598 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:13.692 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:13.695 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:13.873 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:14.167 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-25T10:59:14.293 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-25T10:59:14.349 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2/2 2023-01-25T10:59:14.568 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-25T10:59:14.738 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2/2 2023-01-25T10:59:14.738 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/2 2023-01-25T10:59:14.819 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-25T10:59:15.038 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-25T10:59:15.186 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 13/13 2023-01-25T10:59:15.258 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-25T10:59:15.259 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:15.259 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:15.259 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-25T10:59:15.259 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:15.260 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:15.471 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-volume 2023-01-25T10:59:15.530 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 13/13 2023-01-25T10:59:15.530 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 1/13 2023-01-25T10:59:15.530 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-25T10:59:15.530 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-25T10:59:15.531 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-25T10:59:15.531 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-25T10:59:15.531 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-25T10:59:15.531 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-25T10:59:15.531 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-25T10:59:15.532 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-25T10:59:15.532 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-25T10:59:15.532 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-25T10:59:15.532 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-25T10:59:16.123 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-25T10:59:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:16.124 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:16.124 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-25T10:59:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-25T10:59:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-25T10:59:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-25T10:59:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-25T10:59:16.125 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-25T10:59:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-25T10:59:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-25T10:59:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-25T10:59:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-25T10:59:16.126 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-25T10:59:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-25T10:59:16.127 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:16.127 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:16.205 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:02 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:16.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2023-01-25T10:59:17.012 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:02 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:17.336 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:17.337 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:17.338 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2023-01-25T10:59:17.338 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:17.338 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:17.338 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 263 k 2023-01-25T10:59:17.338 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:17.339 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:17.339 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:17.339 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:59:17.339 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:17.339 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 263 k 2023-01-25T10:59:17.340 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 1.3 M 2023-01-25T10:59:17.340 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:17.612 INFO:teuthology.orchestra.run.smithi080.stdout:ceph-volume-18.0.0-1986.g44e4095d.el8.noarch.rp 962 kB/s | 263 kB 00:00 2023-01-25T10:59:17.613 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:17.613 INFO:teuthology.orchestra.run.smithi080.stdout:Total 958 kB/s | 263 kB 00:00 2023-01-25T10:59:17.613 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:17.622 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:17.623 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:17.707 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:17.710 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:17.889 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:18.175 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:59:18.231 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:18.232 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:18.232 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:59:18.232 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:18.232 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 120 k 2023-01-25T10:59:18.233 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:59:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-25T10:59:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-25T10:59:18.233 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-25T10:59:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-25T10:59:18.234 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:18.234 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:18.234 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:18.234 INFO:teuthology.orchestra.run.smithi027.stdout:Install 5 Packages 2023-01-25T10:59:18.235 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:18.235 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 6.0 M 2023-01-25T10:59:18.235 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2023-01-25T10:59:18.236 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:18.331 INFO:teuthology.orchestra.run.smithi027.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.5 MB/s | 538 kB 00:00 2023-01-25T10:59:18.431 INFO:teuthology.orchestra.run.smithi027.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2023-01-25T10:59:18.482 INFO:teuthology.orchestra.run.smithi027.stdout:(3/5): ceph-mgr-cephadm-18.0.0-1986.g44e4095d.e 485 kB/s | 120 kB 00:00 2023-01-25T10:59:18.607 INFO:teuthology.orchestra.run.smithi027.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 344 kB/s | 60 kB 00:00 2023-01-25T10:59:18.611 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:59:18.757 INFO:teuthology.orchestra.run.smithi027.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.1 MB/s | 476 kB 00:00 2023-01-25T10:59:18.759 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:18.759 INFO:teuthology.orchestra.run.smithi027.stdout:Total 11 MB/s | 6.0 MB 00:00 2023-01-25T10:59:18.812 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:18.823 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:18.824 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:18.963 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:18.967 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:19.107 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:59:19.107 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:19.108 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:19.108 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:19.108 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:19.108 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:19.269 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:19.333 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados-devel 2023-01-25T10:59:19.583 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-25T10:59:20.066 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:06 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:20.615 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-25T10:59:21.006 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-25T10:59:21.203 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:21.204 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:21.205 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2023-01-25T10:59:21.205 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:21.205 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:21.205 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 139 k 2023-01-25T10:59:21.206 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:21.206 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:21.206 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:21.206 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:59:21.206 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:21.206 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 139 k 2023-01-25T10:59:21.207 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 512 k 2023-01-25T10:59:21.207 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:21.265 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-25T10:59:21.408 INFO:teuthology.orchestra.run.smithi080.stdout:librados-devel-18.0.0-1986.g44e4095d.el8.x86_64 686 kB/s | 139 kB 00:00 2023-01-25T10:59:21.409 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:21.409 INFO:teuthology.orchestra.run.smithi080.stdout:Total 682 kB/s | 139 kB 00:00 2023-01-25T10:59:21.410 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:21.414 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:21.414 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:21.452 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 5/5 2023-01-25T10:59:21.467 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:21.470 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:21.595 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:21.756 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 5/5 2023-01-25T10:59:21.757 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 1/5 2023-01-25T10:59:21.757 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-25T10:59:21.757 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-25T10:59:21.757 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-25T10:59:21.788 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:22.089 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:22.283 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-25T10:59:22.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:22.284 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:22.284 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:22.284 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-25T10:59:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-25T10:59:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-25T10:59:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-25T10:59:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:22.285 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:22.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2023-01-25T10:59:22.559 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:22.560 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:22.560 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:22.561 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:22.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs2 2023-01-25T10:59:23.155 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:08 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:23.505 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:09 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:24.385 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:24.385 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:24.386 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:59:24.386 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:24.386 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:24.386 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 845 k 2023-01-25T10:59:24.386 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-01-25T10:59:24.387 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-25T10:59:24.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:24.387 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:24.387 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:24.387 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2023-01-25T10:59:24.388 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:24.388 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 928 k 2023-01-25T10:59:24.388 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.9 M 2023-01-25T10:59:24.388 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:24.456 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-01-25T10:59:24.549 INFO:teuthology.orchestra.run.smithi080.stdout:Package libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:24.648 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:24.649 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:24.649 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:24.690 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-18.0.0-1986.g44e4095d.el8.x86_ 2.7 MB/s | 845 kB 00:00 2023-01-25T10:59:24.691 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:24.691 INFO:teuthology.orchestra.run.smithi027.stdout:Total 3.0 MB/s | 928 kB 00:00 2023-01-25T10:59:24.701 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:24.724 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:24.724 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:24.742 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs-devel 2023-01-25T10:59:24.824 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:24.828 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:25.031 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:25.288 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-25T10:59:25.466 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:11 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:25.484 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2/2 2023-01-25T10:59:25.908 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2/2 2023-01-25T10:59:25.909 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/2 2023-01-25T10:59:26.431 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-25T10:59:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:26.432 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-25T10:59:26.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:26.433 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:26.610 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:26.611 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:26.611 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2023-01-25T10:59:26.612 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:26.612 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:26.612 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 30 k 2023-01-25T10:59:26.612 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:26.613 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:26.613 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:26.613 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:59:26.613 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:26.613 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 30 k 2023-01-25T10:59:26.614 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 137 k 2023-01-25T10:59:26.614 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:26.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-volume 2023-01-25T10:59:26.841 INFO:teuthology.orchestra.run.smithi080.stdout:libcephfs-devel-18.0.0-1986.g44e4095d.el8.x86_6 133 kB/s | 30 kB 00:00 2023-01-25T10:59:26.842 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:26.842 INFO:teuthology.orchestra.run.smithi080.stdout:Total 132 kB/s | 30 kB 00:00 2023-01-25T10:59:26.842 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:26.845 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:26.845 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:26.873 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:26.877 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:26.935 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:27.128 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:27.384 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:12 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:27.388 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:27.825 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:27.825 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:27.826 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:27.826 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:27.826 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:27.826 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:27.939 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados2 2023-01-25T10:59:28.598 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:28.598 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:28.599 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-01-25T10:59:28.599 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:28.599 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:28.599 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:18.0.0-1986.g44e4095d.el8 ceph-noarch 263 k 2023-01-25T10:59:28.599 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:28.600 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:28.600 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:28.600 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:59:28.600 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:28.601 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 263 k 2023-01-25T10:59:28.601 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 1.3 M 2023-01-25T10:59:28.601 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:28.666 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:14 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:28.819 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-volume-18.0.0-1986.g44e4095d.el8.noarch.rp 1.2 MB/s | 263 kB 00:00 2023-01-25T10:59:28.820 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:28.821 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.2 MB/s | 263 kB 00:00 2023-01-25T10:59:28.821 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:28.832 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:28.832 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:28.946 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:28.950 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:29.142 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:29.401 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:59:29.711 INFO:teuthology.orchestra.run.smithi080.stdout:Package librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:29.810 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:29.812 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:29.812 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:29.854 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:59:29.906 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librbd1 2023-01-25T10:59:30.368 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 1/1 2023-01-25T10:59:30.368 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:30.368 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:30.368 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:18.0.0-1986.g44e4095d.el8.noarch 2023-01-25T10:59:30.369 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:30.369 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:30.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2023-01-25T10:59:30.627 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:16 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:31.320 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:16 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:31.671 INFO:teuthology.orchestra.run.smithi080.stdout:Package librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:31.770 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:31.771 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:31.772 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:31.869 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rados 2023-01-25T10:59:32.554 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:32.555 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:32.556 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-25T10:59:32.556 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:32.556 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:32.556 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 139 k 2023-01-25T10:59:32.556 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:32.557 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:32.557 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:32.557 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:59:32.557 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:32.557 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 139 k 2023-01-25T10:59:32.558 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 512 k 2023-01-25T10:59:32.558 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:32.586 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:18 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:32.815 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-18.0.0-1986.g44e4095d.el8.x86_64 537 kB/s | 139 kB 00:00 2023-01-25T10:59:32.816 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:32.816 INFO:teuthology.orchestra.run.smithi027.stdout:Total 535 kB/s | 139 kB 00:00 2023-01-25T10:59:32.816 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:32.822 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:32.822 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:32.880 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:32.884 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:33.050 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:33.620 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:33.720 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:33.721 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:33.721 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:33.816 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rgw 2023-01-25T10:59:34.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:34.530 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:20 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:34.999 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:35.511 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:35.511 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:35.511 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:35.512 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:35.512 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:35.512 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:35.567 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:35.667 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:35.668 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:35.668 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:35.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2023-01-25T10:59:35.774 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-cephfs 2023-01-25T10:59:36.453 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:21 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:36.489 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:22 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:37.537 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:37.553 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:37.636 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:37.637 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:37.637 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:37.663 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:37.665 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:37.665 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:37.740 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rbd 2023-01-25T10:59:37.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2023-01-25T10:59:38.456 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:24 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:38.517 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:23 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:39.487 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:39.588 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:39.589 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2023-01-25T10:59:39.589 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:39.689 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-fuse 2023-01-25T10:59:39.744 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:39.745 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:39.745 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-25T10:59:39.746 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:39.746 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:39.746 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 30 k 2023-01-25T10:59:39.746 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:39.746 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:39.747 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:39.747 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:59:39.747 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 30 k 2023-01-25T10:59:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 137 k 2023-01-25T10:59:39.748 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:40.405 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-18.0.0-1986.g44e4095d.el8.x86_6 46 kB/s | 30 kB 00:00 2023-01-25T10:59:40.405 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:40.406 INFO:teuthology.orchestra.run.smithi027.stdout:Total 46 kB/s | 30 kB 00:00 2023-01-25T10:59:40.406 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:40.411 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:40.411 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:40.422 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:26 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:40.441 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:40.445 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:40.514 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:40.706 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:41.003 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:41.511 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:41.511 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:41.511 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:41.512 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:41.512 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:41.512 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:41.563 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:41.563 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:41.564 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2023-01-25T10:59:41.564 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:41.564 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:41.564 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 94 k 2023-01-25T10:59:41.564 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:41.565 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:41.565 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:41.565 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:59:41.565 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:41.566 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 94 k 2023-01-25T10:59:41.566 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 280 k 2023-01-25T10:59:41.566 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:41.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2023-01-25T10:59:41.896 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-fuse-18.0.0-1986.g44e4095d.el8.x86_64.rpm 285 kB/s | 94 kB 00:00 2023-01-25T10:59:41.897 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:41.897 INFO:teuthology.orchestra.run.smithi080.stdout:Total 284 kB/s | 94 kB 00:00 2023-01-25T10:59:41.897 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:41.906 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:41.907 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:41.957 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:41.960 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:42.070 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:42.389 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:27 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:42.584 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:42.879 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:43.366 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:43.366 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:43.367 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:43.367 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:43.367 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:43.367 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:43.509 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:43.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-mirror 2023-01-25T10:59:43.621 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:43.623 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:43.623 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:43.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2023-01-25T10:59:44.352 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:30 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:44.480 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:29 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:45.486 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:45.487 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:45.487 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2023-01-25T10:59:45.487 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:45.488 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:45.488 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.5 M 2023-01-25T10:59:45.488 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:45.489 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:45.489 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:45.489 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:59:45.490 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:45.490 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.5 M 2023-01-25T10:59:45.490 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 16 M 2023-01-25T10:59:45.491 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:45.595 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:45.708 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:45.710 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:45.710 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:45.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2023-01-25T10:59:45.877 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-mirror-18.0.0-1986.g44e4095d.el8.x86_64.rpm 8.9 MB/s | 3.5 MB 00:00 2023-01-25T10:59:45.877 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:45.877 INFO:teuthology.orchestra.run.smithi080.stdout:Total 8.9 MB/s | 3.5 MB 00:00 2023-01-25T10:59:45.878 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:45.889 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:45.889 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:45.953 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:45.956 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:46.393 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:46.571 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:31 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:46.577 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:47.021 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:47.486 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:47.486 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:47.486 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:47.486 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:47.487 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:47.487 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:47.678 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:47.704 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-nbd 2023-01-25T10:59:47.791 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:47.792 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:47.792 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:47.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2023-01-25T10:59:48.441 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:34 ago on Wed 25 Jan 2023 10:57:14 AM UTC. 2023-01-25T10:59:48.638 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:33 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:49.584 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2023-01-25T10:59:49.585 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:49.585 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2023-01-25T10:59:49.586 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:49.586 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2023-01-25T10:59:49.586 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 180 k 2023-01-25T10:59:49.586 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:49.586 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2023-01-25T10:59:49.587 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2023-01-25T10:59:49.587 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2023-01-25T10:59:49.587 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:49.587 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 180 k 2023-01-25T10:59:49.587 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 568 k 2023-01-25T10:59:49.588 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2023-01-25T10:59:49.746 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:49.857 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:49.858 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:49.859 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:49.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2023-01-25T10:59:49.988 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-nbd-18.0.0-1986.g44e4095d.el8.x86_64.rpm 447 kB/s | 180 kB 00:00 2023-01-25T10:59:49.988 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:49.989 INFO:teuthology.orchestra.run.smithi080.stdout:Total 446 kB/s | 180 kB 00:00 2023-01-25T10:59:49.989 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2023-01-25T10:59:50.000 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2023-01-25T10:59:50.000 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2023-01-25T10:59:50.052 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2023-01-25T10:59:50.055 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2023-01-25T10:59:50.169 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2023-01-25T10:59:50.374 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:50.682 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:50.712 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:35 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:51.156 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2023-01-25T10:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T10:59:51.157 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2023-01-25T10:59:51.354 DEBUG:teuthology.parallel:result is None 2023-01-25T10:59:51.830 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:51.942 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:51.943 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:51.943 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:52.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2023-01-25T10:59:52.786 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:37 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:53.927 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:18.0.0-1986.g44e4095d.el8.x86_64 is already installed. 2023-01-25T10:59:54.042 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-01-25T10:59:54.044 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:54.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2023-01-25T10:59:54.903 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:39 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:56.130 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:56.131 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:56.131 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-01-25T10:59:56.131 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:56.131 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:56.132 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 94 k 2023-01-25T10:59:56.132 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:56.132 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:56.132 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:56.132 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:59:56.133 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:56.133 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 94 k 2023-01-25T10:59:56.133 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 280 k 2023-01-25T10:59:56.133 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T10:59:56.336 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-18.0.0-1986.g44e4095d.el8.x86_64.rpm 461 kB/s | 94 kB 00:00 2023-01-25T10:59:56.337 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T10:59:56.337 INFO:teuthology.orchestra.run.smithi027.stdout:Total 458 kB/s | 94 kB 00:00 2023-01-25T10:59:56.338 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T10:59:56.349 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T10:59:56.349 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T10:59:56.404 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T10:59:56.408 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T10:59:56.526 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T10:59:56.729 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:57.568 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T10:59:57.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:57.569 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T10:59:57.569 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T10:59:57.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:57.569 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T10:59:57.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2023-01-25T10:59:58.545 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:43 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T10:59:59.770 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T10:59:59.771 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:59.772 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-01-25T10:59:59.772 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:59.772 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T10:59:59.772 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 3.5 M 2023-01-25T10:59:59.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:59.773 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T10:59:59.773 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T10:59:59.773 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T10:59:59.773 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T10:59:59.774 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.5 M 2023-01-25T10:59:59.774 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 16 M 2023-01-25T10:59:59.774 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T11:00:01.256 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-18.0.0-1986.g44e4095d.el8.x86_64.rpm 2.3 MB/s | 3.5 MB 00:01 2023-01-25T11:00:01.256 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T11:00:01.257 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.3 MB/s | 3.5 MB 00:01 2023-01-25T11:00:01.257 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T11:00:01.271 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T11:00:01.271 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T11:00:01.344 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T11:00:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T11:00:01.906 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T11:00:02.073 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T11:00:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T11:00:03.078 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T11:00:03.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:03.079 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T11:00:03.080 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T11:00:03.080 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:03.080 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T11:00:03.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2023-01-25T11:00:04.053 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:49 ago on Wed 25 Jan 2023 10:57:15 AM UTC. 2023-01-25T11:00:05.271 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-01-25T11:00:05.272 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T11:00:05.273 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-01-25T11:00:05.273 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T11:00:05.273 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-01-25T11:00:05.273 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:18.0.0-1986.g44e4095d.el8 ceph 180 k 2023-01-25T11:00:05.273 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:05.274 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-01-25T11:00:05.274 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-01-25T11:00:05.274 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-01-25T11:00:05.274 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:05.275 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 180 k 2023-01-25T11:00:05.275 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 568 k 2023-01-25T11:00:05.275 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-01-25T11:00:05.483 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-18.0.0-1986.g44e4095d.el8.x86_64.rpm 856 kB/s | 180 kB 00:00 2023-01-25T11:00:05.484 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-01-25T11:00:05.484 INFO:teuthology.orchestra.run.smithi027.stdout:Total 851 kB/s | 180 kB 00:00 2023-01-25T11:00:05.485 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-01-25T11:00:05.497 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-01-25T11:00:05.497 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-01-25T11:00:05.553 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-01-25T11:00:05.556 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-01-25T11:00:05.676 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-01-25T11:00:05.836 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T11:00:06.150 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T11:00:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 1/1 2023-01-25T11:00:06.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:06.654 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-01-25T11:00:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:18.0.0-1986.g44e4095d.el8.x86_64 2023-01-25T11:00:06.654 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:06.654 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-01-25T11:00:06.868 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:06.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo lsb_release -is 2023-01-25T11:00:06.907 INFO:teuthology.orchestra.run.smithi027.stdout:CentOSStream 2023-01-25T11:00:06.908 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T11:00:06.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T11:00:07.067 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-25T11:00:07.091 INFO:teuthology.orchestra.run.smithi027.stdout:18.0.0-1986.g44e4095d.el8 2023-01-25T11:00:07.092 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1986.g44e4095d.el8 2023-01-25T11:00:07.092 INFO:teuthology.task.install:The correct ceph version 18.0.0-1986.g44e4095d is installed. 2023-01-25T11:00:07.093 DEBUG:teuthology.orchestra.run.smithi080:> sudo lsb_release -is 2023-01-25T11:00:07.135 INFO:teuthology.orchestra.run.smithi080.stdout:CentOSStream 2023-01-25T11:00:07.136 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T11:00:07.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=44e4095da12a8269eb6cfe163cbb0d00b1a6412c 2023-01-25T11:00:07.296 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-25T11:00:07.320 INFO:teuthology.orchestra.run.smithi080.stdout:18.0.0-1986.g44e4095d.el8 2023-01-25T11:00:07.321 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1986.g44e4095d.el8 2023-01-25T11:00:07.321 INFO:teuthology.task.install:The correct ceph version 18.0.0-1986.g44e4095d is installed. 2023-01-25T11:00:07.322 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-25T11:00:07.322 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:07.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-25T11:00:07.352 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:07.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-25T11:00:07.389 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-25T11:00:07.390 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:07.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2023-01-25T11:00:07.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-25T11:00:07.489 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:07.489 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2023-01-25T11:00:07.517 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-25T11:00:07.585 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-25T11:00:07.586 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:07.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-25T11:00:07.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-25T11:00:07.682 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:07.682 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-25T11:00:07.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-25T11:00:07.777 INFO:teuthology.run_tasks:Running task ceph... 2023-01-25T11:00:07.843 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-25T11:00:07.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2023-01-25T11:00:07.846 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /var/log/ceph 2023-01-25T11:00:07.875 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-25T11:00:07.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-25T11:00:07.915 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-25T11:00:07.941 INFO:tasks.ceph:Creating extra log directories... 2023-01-25T11:00:07.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-25T11:00:07.983 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-25T11:00:08.023 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-25T11:00:08.023 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-25T11:00:08.024 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137291', 'branch': 'wip-yuri6-testing-2023-01-24-0746', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench}', 'email': 'yweinstein@radhat.com', 'first_in_suite': False, 'job_id': '7137291', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2023-01-24-0746'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2023-01-24-0746', 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-yuri6-testing-2023-01-24-0746', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_44e4095da12a8269eb6cfe163cbb0d00b1a6412c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c', 'targets': {'smithi027.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTYuTo/n+C846Gze/yaHvARXHI539iadzzB7+MPcOBEkSQase84qfMhMp2JMV75DGixGl3+NCNkKjltDitSxSL3yXd6aQFsnpjdBKjVAGm0Nst4U2PuFJBYwgk6K5sgZ/TrNiCby8o2y9UILOBDvm42heaNZ57hJKGd2uihKfr+7EADgGQUL8r7TdZ4Jpv7QiqL8fS/hp3MdUtYJUUvNXdrgke7azIp0xFNe9Tra+RIUprs/XNrFxPghLhPE5d8Ew71IlLfXiPkovAZS80TeArpCrpd5fVzRTpfpSIhncsS4lU/Vx6HTtQDqpBxZ5ZUMDzEntzn60xRGVCpk5FgbX/QzQxS+2aPda7HiYVunglfVPf6cDV0XRbxfNi7xuupblWZv6rRfDzxyfWTrB8GAwvlZ0Dp93zdSDOAL9EFbb3qLZ+bfEPA3fkkJDxXonBdPeauUtlkWR9uMSfNBJ1gMRYacivuyBPAdvWvbWKnvBiBdl+tUokwLaRtdqaJA+E5bE=', 'smithi080.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIHDBU5pLgczGe7/SMByZJ3XQqWu8JfzaM/bFu6xnzpvtzD6uSQUZsXA4ObSmfM0fEX3zuJP9JvD5nYQFm9kWbvjtG/2unY/xHZ1MCMAtdvRjnsn7YnCupDU4b1nTYNx2mFCJgouRa7SsxuhIt31RUDC4zndu+VN+elEJ8UTgX9hOV3qYKlZhVkpBlmW2DwAgUjmt83xvCoYYNdu9hzYza/kOBBg6OuRuw7ny/ZxIY1rWqvlUdOhldcQlCoRdRvMftNMDsWCuPhXoBUIZhykdGML5lhkqHzOKU9P6DphNTWn6f48jHxALfkpDHCmRN8huc+FS35ly4BKlEfzgprUSRme+88RzjojOqb9kdJJelLuGyK7WUX7WlV+yvU4ovpaOv6c/2NXtwiGLeiX86PXWxQ/h38EC/YiAT92vFst+PEQws6vbiRz2rcO3MiAe2NP8ZsiE8LC2JLnrQ20CJHzQ++C2VNVUQjMHaRHeEYxhtweUhZ9Rz9IKysiecRp/GQvc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '44e4095da12a8269eb6cfe163cbb0d00b1a6412c', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_22:27:30', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-25T11:00:08.025 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-25T11:00:08.052 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-25T11:00:08.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2023-01-25T11:00:08.110 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/run/ceph 2023-01-25T11:00:08.137 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:08.138 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2023-01-25T11:00:08.193 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-25T11:00:08.194 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2023-01-25T11:00:08.252 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-25T11:00:08.252 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:08.252 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 372886 Links: 1 2023-01-25T11:00:08.252 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:08.253 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:08.253 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-25 11:00:02.273635726 +0000 2023-01-25T11:00:08.253 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-25 10:56:26.590487937 +0000 2023-01-25T11:00:08.254 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-25 10:56:26.590487937 +0000 2023-01-25T11:00:08.254 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-25T11:00:08.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-25T11:00:08.318 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-25T11:00:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-25T11:00:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000226214 s, 2.3 MB/s 2023-01-25T11:00:08.321 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-25T11:00:08.382 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2023-01-25T11:00:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-25T11:00:08.441 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:08.441 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 371453 Links: 1 2023-01-25T11:00:08.442 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:08.442 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:08.442 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-25 11:00:02.273635726 +0000 2023-01-25T11:00:08.442 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-25 10:56:27.103475130 +0000 2023-01-25T11:00:08.442 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-25 10:56:27.103475130 +0000 2023-01-25T11:00:08.443 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-25T11:00:08.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-25T11:00:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-25T11:00:08.508 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-25T11:00:08.509 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000210946 s, 2.4 MB/s 2023-01-25T11:00:08.510 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-25T11:00:08.569 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2023-01-25T11:00:08.627 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-25T11:00:08.628 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:08.628 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 370343 Links: 1 2023-01-25T11:00:08.628 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:08.628 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:08.629 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-25 11:00:02.273635726 +0000 2023-01-25T11:00:08.629 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-25 10:56:27.619462249 +0000 2023-01-25T11:00:08.629 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-25 10:56:27.619462249 +0000 2023-01-25T11:00:08.629 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-25T11:00:08.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-25T11:00:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-25T11:00:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-25T11:00:08.694 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000209449 s, 2.4 MB/s 2023-01-25T11:00:08.695 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-25T11:00:08.754 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2023-01-25T11:00:08.813 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-25T11:00:08.813 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:08.814 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 371561 Links: 1 2023-01-25T11:00:08.814 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:08.814 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:08.814 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-01-25 11:00:02.273635726 +0000 2023-01-25T11:00:08.814 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-01-25 10:56:28.157448818 +0000 2023-01-25T11:00:08.815 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-01-25 10:56:28.157448818 +0000 2023-01-25T11:00:08.815 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-01-25T11:00:08.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-25T11:00:08.880 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-01-25T11:00:08.880 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-01-25T11:00:08.880 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000183888 s, 2.8 MB/s 2023-01-25T11:00:08.881 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-25T11:00:08.940 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:08.940 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:08.941 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2023-01-25T11:00:08.960 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-25T11:00:08.960 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 2023-01-25T11:00:09.018 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-25T11:00:09.019 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:09.019 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 372125 Links: 1 2023-01-25T11:00:09.019 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:09.019 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:09.020 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2023-01-25 10:59:46.760818372 +0000 2023-01-25T11:00:09.020 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2023-01-25 10:56:18.589500744 +0000 2023-01-25T11:00:09.020 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2023-01-25 10:56:18.589500744 +0000 2023-01-25T11:00:09.020 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2023-01-25T11:00:09.021 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-25T11:00:09.086 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2023-01-25T11:00:09.086 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2023-01-25T11:00:09.087 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000265344 s, 1.9 MB/s 2023-01-25T11:00:09.088 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-25T11:00:09.147 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 2023-01-25T11:00:09.205 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-25T11:00:09.205 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:09.205 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 376376 Links: 1 2023-01-25T11:00:09.205 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:09.206 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:09.206 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2023-01-25 10:59:46.760818372 +0000 2023-01-25T11:00:09.206 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2023-01-25 10:56:19.099489668 +0000 2023-01-25T11:00:09.206 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2023-01-25 10:56:19.099489668 +0000 2023-01-25T11:00:09.206 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2023-01-25T11:00:09.207 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-25T11:00:09.271 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2023-01-25T11:00:09.271 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2023-01-25T11:00:09.271 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000218657 s, 2.3 MB/s 2023-01-25T11:00:09.272 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-25T11:00:09.330 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 2023-01-25T11:00:09.389 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-25T11:00:09.389 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:09.389 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 371418 Links: 1 2023-01-25T11:00:09.389 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:09.390 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:09.390 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2023-01-25 10:59:46.760818372 +0000 2023-01-25T11:00:09.390 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2023-01-25 10:56:19.608478615 +0000 2023-01-25T11:00:09.390 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2023-01-25 10:56:19.608478615 +0000 2023-01-25T11:00:09.390 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2023-01-25T11:00:09.391 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-25T11:00:09.455 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2023-01-25T11:00:09.455 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2023-01-25T11:00:09.455 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000228203 s, 2.2 MB/s 2023-01-25T11:00:09.456 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-25T11:00:09.516 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 2023-01-25T11:00:09.573 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-25T11:00:09.574 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T11:00:09.574 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 375433 Links: 1 2023-01-25T11:00:09.574 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T11:00:09.574 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T11:00:09.574 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2023-01-25 10:59:46.761818350 +0000 2023-01-25T11:00:09.575 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2023-01-25 10:56:20.099467953 +0000 2023-01-25T11:00:09.575 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2023-01-25 10:56:20.099467953 +0000 2023-01-25T11:00:09.575 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2023-01-25T11:00:09.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-25T11:00:09.638 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2023-01-25T11:00:09.639 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2023-01-25T11:00:09.639 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000186108 s, 2.8 MB/s 2023-01-25T11:00:09.640 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-25T11:00:09.699 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:09.699 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-01-25T11:00:09.699 INFO:tasks.ceph:Generating config... 2023-01-25T11:00:09.701 INFO:tasks.ceph:[osd] debug monc = 20 2023-01-25T11:00:09.701 INFO:tasks.ceph:[osd] bdev async discard = True 2023-01-25T11:00:09.701 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-01-25T11:00:09.701 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-01-25T11:00:09.702 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-01-25T11:00:09.702 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-01-25T11:00:09.702 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-01-25T11:00:09.703 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-01-25T11:00:09.703 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-25T11:00:09.703 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-25T11:00:09.703 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-01-25T11:00:09.704 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-01-25T11:00:09.704 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-01-25T11:00:09.705 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-01-25T11:00:09.705 INFO:tasks.ceph:[osd] osd backoff on peering = True 2023-01-25T11:00:09.705 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-01-25T11:00:09.705 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-01-25T11:00:09.706 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-01-25T11:00:09.706 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-01-25T11:00:09.706 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2023-01-25T11:00:09.706 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-01-25T11:00:09.707 INFO:tasks.ceph:[osd] osd max scrubs = 2 2023-01-25T11:00:09.707 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-01-25T11:00:09.707 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-01-25T11:00:09.707 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-01-25T11:00:09.708 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-01-25T11:00:09.709 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-01-25T11:00:09.709 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-25T11:00:09.709 INFO:tasks.ceph:[client.0] debug ms = 1 2023-01-25T11:00:09.710 INFO:tasks.ceph:[client.0] debug objecter = 20 2023-01-25T11:00:09.710 INFO:tasks.ceph:[client.0] debug rados = 20 2023-01-25T11:00:09.710 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-01-25T11:00:09.711 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-01-25T11:00:09.711 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-01-25T11:00:09.711 INFO:tasks.ceph:[global] ms type = async 2023-01-25T11:00:09.711 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2023-01-25T11:00:09.712 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2023-01-25T11:00:09.712 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-01-25T11:00:09.712 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2023-01-25T11:00:09.713 INFO:tasks.ceph:[global] osd_recovery_max_active = 10 2023-01-25T11:00:09.713 INFO:tasks.ceph:[global] osd_recovery_max_single_start = 10 2023-01-25T11:00:09.713 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-25T11:00:09.714 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-25T11:00:09.714 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-25T11:00:09.714 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-25T11:00:09.715 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-25T11:00:09.715 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-01-25T11:00:09.715 INFO:tasks.ceph:Setting up mon.a... 2023-01-25T11:00:09.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-25T11:00:09.771 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2023-01-25T11:00:09.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-25T11:00:09.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-25T11:00:09.849 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.c', '[v2:172.21.15.27:3301,v1:172.21.15.27:6790]'), ('mon.b', '172.21.15.80')] 2023-01-25T11:00:09.849 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,[v2:172.21.15.27:3301,v1:172.21.15.27:6790],172.21.15.80', 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}} 2023-01-25T11:00:09.850 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:09.851 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-25T11:00:09.908 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --addv c '[v2:172.21.15.27:3301,v1:172.21.15.27:6790]' --add b 172.21.15.80 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:09.998 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:09.999 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid be11100e-61e6-4b91-8166-d134ac9fbd8b 2023-01-25T11:00:09.999 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = pacific 2023-01-25T11:00:09.999 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2023-01-25T11:00:09.999 INFO:teuthology.orchestra.run.smithi027.stdout:fsid be11100e-61e6-4b91-8166-d134ac9fbd8b 2023-01-25T11:00:09.999 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2023-01-25T11:00:09.997193+0000 2023-01-25T11:00:10.000 INFO:teuthology.orchestra.run.smithi027.stdout:created 2023-01-25T11:00:09.997193+0000 2023-01-25T11:00:10.000 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 16 (pacific) 2023-01-25T11:00:10.000 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2023-01-25T11:00:10.000 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2023-01-25T11:00:10.000 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.b 2023-01-25T11:00:10.001 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] mon.c 2023-01-25T11:00:10.001 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-01-25T11:00:10.002 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-25T11:00:10.020 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID be11100e-61e6-4b91-8166-d134ac9fbd8b... 2023-01-25T11:00:10.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-25T11:00:10.063 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-25T11:00:10.107 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2023-01-25T11:00:10.108 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2023-01-25T11:00:10.108 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-25T11:00:10.108 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2023-01-25T11:00:10.108 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.109 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2023-01-25T11:00:10.109 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.109 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2023-01-25T11:00:10.110 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.110 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2023-01-25T11:00:10.110 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2023-01-25T11:00:10.110 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.110 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2023-01-25T11:00:10.111 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2023-01-25T11:00:10.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.111 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-25T11:00:10.111 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2023-01-25T11:00:10.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.112 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2023-01-25T11:00:10.112 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-25T11:00:10.112 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.112 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2023-01-25T11:00:10.112 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.113 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2023-01-25T11:00:10.113 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2023-01-25T11:00:10.113 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2023-01-25T11:00:10.113 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2023-01-25T11:00:10.113 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2023-01-25T11:00:10.114 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2023-01-25T11:00:10.114 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2023-01-25T11:00:10.114 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-25T11:00:10.114 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-25T11:00:10.114 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2023-01-25T11:00:10.115 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.115 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-25T11:00:10.115 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.115 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2023-01-25T11:00:10.115 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.116 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2023-01-25T11:00:10.116 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.116 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2023-01-25T11:00:10.116 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2023-01-25T11:00:10.117 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2023-01-25T11:00:10.117 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,[v2:172.21.15.27:3301,v1:172.21.15.27:6790],172.21.15.80" 2023-01-25T11:00:10.117 INFO:teuthology.orchestra.run.smithi027.stdout: mon election default strategy = 1 2023-01-25T11:00:10.117 INFO:teuthology.orchestra.run.smithi027.stdout: ms bind msgr1 = False 2023-01-25T11:00:10.118 INFO:teuthology.orchestra.run.smithi027.stdout: ms bind msgr2 = True 2023-01-25T11:00:10.118 INFO:teuthology.orchestra.run.smithi027.stdout: ms type = async 2023-01-25T11:00:10.118 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug inject dispatch delay duration = 0.1 2023-01-25T11:00:10.118 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug inject dispatch delay probability = 0.1 2023-01-25T11:00:10.119 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_min_size = 2 2023-01-25T11:00:10.119 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_size = 2 2023-01-25T11:00:10.119 INFO:teuthology.orchestra.run.smithi027.stdout: osd_recovery_max_active = 10 2023-01-25T11:00:10.119 INFO:teuthology.orchestra.run.smithi027.stdout: osd_recovery_max_single_start = 10 2023-01-25T11:00:10.119 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = be11100e-61e6-4b91-8166-d134ac9fbd8b 2023-01-25T11:00:10.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.120 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2023-01-25T11:00:10.120 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2023-01-25T11:00:10.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.120 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2023-01-25T11:00:10.121 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2023-01-25T11:00:10.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.121 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2023-01-25T11:00:10.121 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2023-01-25T11:00:10.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2023-01-25T11:00:10.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2023-01-25T11:00:10.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2023-01-25T11:00:10.122 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2023-01-25T11:00:10.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2023-01-25T11:00:10.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2023-01-25T11:00:10.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2023-01-25T11:00:10.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.124 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2023-01-25T11:00:10.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.124 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2023-01-25T11:00:10.125 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2023-01-25T11:00:10.125 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.125 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2023-01-25T11:00:10.125 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2023-01-25T11:00:10.125 INFO:teuthology.orchestra.run.smithi027.stdout: debug monc = 20 2023-01-25T11:00:10.126 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2023-01-25T11:00:10.126 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2023-01-25T11:00:10.126 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2023-01-25T11:00:10.126 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2023-01-25T11:00:10.126 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore zero block detection = True 2023-01-25T11:00:10.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 20 2023-01-25T11:00:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 20 2023-01-25T11:00:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-25T11:00:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2023-01-25T11:00:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 10 2023-01-25T11:00:10.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2023-01-25T11:00:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2023-01-25T11:00:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2023-01-25T11:00:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: osd backoff on peering = True 2023-01-25T11:00:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: osd blocked scrub grace period = 3600 2023-01-25T11:00:10.128 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify cached snaps = True 2023-01-25T11:00:10.129 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify missing on start = True 2023-01-25T11:00:10.129 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2023-01-25T11:00:10.129 INFO:teuthology.orchestra.run.smithi027.stdout: osd inject bad map crc probability = 0.1 2023-01-25T11:00:10.129 INFO:teuthology.orchestra.run.smithi027.stdout: osd max markdown count = 1000 2023-01-25T11:00:10.129 INFO:teuthology.orchestra.run.smithi027.stdout: osd max scrubs = 2 2023-01-25T11:00:10.130 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock override recovery settings = True 2023-01-25T11:00:10.130 INFO:teuthology.orchestra.run.smithi027.stdout: osd memory target = 1610612736 2023-01-25T11:00:10.130 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2023-01-25T11:00:10.130 INFO:teuthology.orchestra.run.smithi027.stdout: osd op queue = debug_random 2023-01-25T11:00:10.130 INFO:teuthology.orchestra.run.smithi027.stdout: osd op queue cut off = debug_random 2023-01-25T11:00:10.131 INFO:teuthology.orchestra.run.smithi027.stdout: osd shutdown pgref assert = True 2023-01-25T11:00:10.131 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.131 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2023-01-25T11:00:10.131 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-25T11:00:10.131 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2023-01-25T11:00:10.132 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-01-25T11:00:10.132 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-01-25T11:00:10.132 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-01-25T11:00:10.132 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-01-25T11:00:10.132 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2023-01-25T11:00:10.133 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.133 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2023-01-25T11:00:10.133 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-25T11:00:10.134 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-01-25T11:00:10.134 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2023-01-25T11:00:10.134 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-01-25T11:00:10.135 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2023-01-25T11:00:10.135 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2023-01-25T11:00:10.135 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-01-25T11:00:10.136 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2023-01-25T11:00:10.136 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2023-01-25T11:00:10.136 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-01-25T11:00:10.137 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.137 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-25T11:00:10.137 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2023-01-25T11:00:10.137 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2023-01-25T11:00:10.137 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.138 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2023-01-25T11:00:10.138 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-25T11:00:10.138 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-25T11:00:10.138 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.138 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2023-01-25T11:00:10.139 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2023-01-25T11:00:10.139 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.139 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2023-01-25T11:00:10.139 INFO:teuthology.orchestra.run.smithi027.stdout: mon scrub interval = 300 2023-01-25T11:00:10.139 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:10.140 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2023-01-25T11:00:10.140 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2023-01-25T11:00:10.140 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2023-01-25T11:00:10.140 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2023-01-25T11:00:10.140 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-25T11:00:10.141 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-25T11:00:10.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2023-01-25T11:00:10.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2023-01-25T11:00:10.141 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2023-01-25T11:00:10.141 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2023-01-25T11:00:10.141 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-01-25T11:00:10.142 INFO:teuthology.orchestra.run.smithi027.stdout: debug objecter = 20 2023-01-25T11:00:10.142 INFO:teuthology.orchestra.run.smithi027.stdout: debug rados = 20 2023-01-25T11:00:10.143 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2023-01-25T11:00:10.144 INFO:teuthology.orchestra.run.smithi080.stdout: chdir = "" 2023-01-25T11:00:10.144 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-25T11:00:10.144 INFO:teuthology.orchestra.run.smithi080.stdout: auth supported = cephx 2023-01-25T11:00:10.144 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.145 INFO:teuthology.orchestra.run.smithi080.stdout: filestore xattr use omap = true 2023-01-25T11:00:10.145 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.145 INFO:teuthology.orchestra.run.smithi080.stdout: mon clock drift allowed = 1.000 2023-01-25T11:00:10.145 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.145 INFO:teuthology.orchestra.run.smithi080.stdout: osd crush chooseleaf type = 0 2023-01-25T11:00:10.146 INFO:teuthology.orchestra.run.smithi080.stdout: auth debug = true 2023-01-25T11:00:10.146 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.146 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on old message = true 2023-01-25T11:00:10.146 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on bug = true 2023-01-25T11:00:10.146 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.147 INFO:teuthology.orchestra.run.smithi080.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-25T11:00:10.147 INFO:teuthology.orchestra.run.smithi080.stdout: mon pg warn max object skew = 0 2023-01-25T11:00:10.147 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.147 INFO:teuthology.orchestra.run.smithi080.stdout: # disable pg_autoscaler by default for new pools 2023-01-25T11:00:10.147 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-25T11:00:10.148 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.148 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default size = 2 2023-01-25T11:00:10.148 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.148 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow primary affinity = true 2023-01-25T11:00:10.148 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow pg remap = true 2023-01-25T11:00:10.149 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on legacy crush tunables = false 2023-01-25T11:00:10.149 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on crush straw calc version zero = false 2023-01-25T11:00:10.149 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on no sortbitwise = false 2023-01-25T11:00:10.149 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on osd down out interval zero = false 2023-01-25T11:00:10.149 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on too few osds = false 2023-01-25T11:00:10.150 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-25T11:00:10.150 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-25T11:00:10.150 INFO:teuthology.orchestra.run.smithi080.stdout: mon_allow_pool_size_one = true 2023-01-25T11:00:10.150 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.150 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-25T11:00:10.151 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.151 INFO:teuthology.orchestra.run.smithi080.stdout: osd default data pool replay window = 5 2023-01-25T11:00:10.151 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.151 INFO:teuthology.orchestra.run.smithi080.stdout: mon allow pool delete = true 2023-01-25T11:00:10.151 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.152 INFO:teuthology.orchestra.run.smithi080.stdout: mon cluster log file level = debug 2023-01-25T11:00:10.152 INFO:teuthology.orchestra.run.smithi080.stdout: debug asserts on shutdown = true 2023-01-25T11:00:10.152 INFO:teuthology.orchestra.run.smithi080.stdout: mon health detail to clog = false 2023-01-25T11:00:10.152 INFO:teuthology.orchestra.run.smithi080.stdout: mon host = "172.21.15.27,[v2:172.21.15.27:3301,v1:172.21.15.27:6790],172.21.15.80" 2023-01-25T11:00:10.153 INFO:teuthology.orchestra.run.smithi080.stdout: mon election default strategy = 1 2023-01-25T11:00:10.153 INFO:teuthology.orchestra.run.smithi080.stdout: ms bind msgr1 = False 2023-01-25T11:00:10.153 INFO:teuthology.orchestra.run.smithi080.stdout: ms bind msgr2 = True 2023-01-25T11:00:10.153 INFO:teuthology.orchestra.run.smithi080.stdout: ms type = async 2023-01-25T11:00:10.153 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug inject dispatch delay duration = 0.1 2023-01-25T11:00:10.154 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug inject dispatch delay probability = 0.1 2023-01-25T11:00:10.154 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_min_size = 2 2023-01-25T11:00:10.154 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_size = 2 2023-01-25T11:00:10.154 INFO:teuthology.orchestra.run.smithi080.stdout: osd_recovery_max_active = 10 2023-01-25T11:00:10.154 INFO:teuthology.orchestra.run.smithi080.stdout: osd_recovery_max_single_start = 10 2023-01-25T11:00:10.155 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = be11100e-61e6-4b91-8166-d134ac9fbd8b 2023-01-25T11:00:10.155 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.155 INFO:teuthology.orchestra.run.smithi080.stdout:[osd] 2023-01-25T11:00:10.155 INFO:teuthology.orchestra.run.smithi080.stdout: osd journal size = 100 2023-01-25T11:00:10.155 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.155 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub load threshold = 5.0 2023-01-25T11:00:10.156 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub max interval = 600 2023-01-25T11:00:10.156 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.156 INFO:teuthology.orchestra.run.smithi080.stdout: osd recover clone overlap = true 2023-01-25T11:00:10.156 INFO:teuthology.orchestra.run.smithi080.stdout: osd recovery max chunk = 1048576 2023-01-25T11:00:10.156 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.157 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug shutdown = true 2023-01-25T11:00:10.157 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug op order = true 2023-01-25T11:00:10.157 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify stray on activate = true 2023-01-25T11:00:10.157 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.157 INFO:teuthology.orchestra.run.smithi080.stdout: osd open classes on start = true 2023-01-25T11:00:10.158 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug pg log writeout = true 2023-01-25T11:00:10.158 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.158 INFO:teuthology.orchestra.run.smithi080.stdout: osd deep scrub update digest min age = 30 2023-01-25T11:00:10.158 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.158 INFO:teuthology.orchestra.run.smithi080.stdout: osd map max advance = 10 2023-01-25T11:00:10.159 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.159 INFO:teuthology.orchestra.run.smithi080.stdout: journal zero on create = true 2023-01-25T11:00:10.159 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.159 INFO:teuthology.orchestra.run.smithi080.stdout: filestore ondisk finisher threads = 3 2023-01-25T11:00:10.159 INFO:teuthology.orchestra.run.smithi080.stdout: filestore apply finisher threads = 3 2023-01-25T11:00:10.160 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.160 INFO:teuthology.orchestra.run.smithi080.stdout: bdev debug aio = true 2023-01-25T11:00:10.160 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug misdirected ops = true 2023-01-25T11:00:10.160 INFO:teuthology.orchestra.run.smithi080.stdout: debug monc = 20 2023-01-25T11:00:10.160 INFO:teuthology.orchestra.run.smithi080.stdout: bdev async discard = True 2023-01-25T11:00:10.161 INFO:teuthology.orchestra.run.smithi080.stdout: bdev enable discard = True 2023-01-25T11:00:10.161 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore block size = 96636764160 2023-01-25T11:00:10.161 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore fsck on mount = True 2023-01-25T11:00:10.161 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore zero block detection = True 2023-01-25T11:00:10.161 INFO:teuthology.orchestra.run.smithi080.stdout: debug bluefs = 20 2023-01-25T11:00:10.162 INFO:teuthology.orchestra.run.smithi080.stdout: debug bluestore = 20 2023-01-25T11:00:10.162 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2023-01-25T11:00:10.162 INFO:teuthology.orchestra.run.smithi080.stdout: debug osd = 20 2023-01-25T11:00:10.162 INFO:teuthology.orchestra.run.smithi080.stdout: debug rocksdb = 10 2023-01-25T11:00:10.162 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd backfillfull_ratio = 0.85 2023-01-25T11:00:10.162 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd full ratio = 0.9 2023-01-25T11:00:10.163 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd nearfull ratio = 0.8 2023-01-25T11:00:10.163 INFO:teuthology.orchestra.run.smithi080.stdout: osd backoff on peering = True 2023-01-25T11:00:10.163 INFO:teuthology.orchestra.run.smithi080.stdout: osd blocked scrub grace period = 3600 2023-01-25T11:00:10.163 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify cached snaps = True 2023-01-25T11:00:10.164 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify missing on start = True 2023-01-25T11:00:10.164 INFO:teuthology.orchestra.run.smithi080.stdout: osd failsafe full ratio = 0.95 2023-01-25T11:00:10.164 INFO:teuthology.orchestra.run.smithi080.stdout: osd inject bad map crc probability = 0.1 2023-01-25T11:00:10.165 INFO:teuthology.orchestra.run.smithi080.stdout: osd max markdown count = 1000 2023-01-25T11:00:10.165 INFO:teuthology.orchestra.run.smithi080.stdout: osd max scrubs = 2 2023-01-25T11:00:10.165 INFO:teuthology.orchestra.run.smithi080.stdout: osd mclock override recovery settings = True 2023-01-25T11:00:10.166 INFO:teuthology.orchestra.run.smithi080.stdout: osd memory target = 1610612736 2023-01-25T11:00:10.166 INFO:teuthology.orchestra.run.smithi080.stdout: osd objectstore = bluestore 2023-01-25T11:00:10.166 INFO:teuthology.orchestra.run.smithi080.stdout: osd op queue = debug_random 2023-01-25T11:00:10.167 INFO:teuthology.orchestra.run.smithi080.stdout: osd op queue cut off = debug_random 2023-01-25T11:00:10.167 INFO:teuthology.orchestra.run.smithi080.stdout: osd shutdown pgref assert = True 2023-01-25T11:00:10.168 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.168 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr] 2023-01-25T11:00:10.168 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2023-01-25T11:00:10.169 INFO:teuthology.orchestra.run.smithi080.stdout: debug mgr = 20 2023-01-25T11:00:10.169 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2023-01-25T11:00:10.169 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2023-01-25T11:00:10.170 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2023-01-25T11:00:10.170 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2023-01-25T11:00:10.170 INFO:teuthology.orchestra.run.smithi080.stdout: mgr/telemetry/nag = false 2023-01-25T11:00:10.171 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.171 INFO:teuthology.orchestra.run.smithi080.stdout:[mon] 2023-01-25T11:00:10.171 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2023-01-25T11:00:10.172 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2023-01-25T11:00:10.172 INFO:teuthology.orchestra.run.smithi080.stdout: debug paxos = 20 2023-01-25T11:00:10.172 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2023-01-25T11:00:10.172 INFO:teuthology.orchestra.run.smithi080.stdout: mon data avail warn = 5 2023-01-25T11:00:10.173 INFO:teuthology.orchestra.run.smithi080.stdout: mon mgr mkfs grace = 240 2023-01-25T11:00:10.173 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2023-01-25T11:00:10.173 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd reporter subtree level = osd 2023-01-25T11:00:10.173 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd prime pg temp = true 2023-01-25T11:00:10.173 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2023-01-25T11:00:10.174 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.174 INFO:teuthology.orchestra.run.smithi080.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-25T11:00:10.174 INFO:teuthology.orchestra.run.smithi080.stdout: auth mon ticket ttl = 660 # 11m 2023-01-25T11:00:10.174 INFO:teuthology.orchestra.run.smithi080.stdout: auth service ticket ttl = 240 # 4m 2023-01-25T11:00:10.174 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.175 INFO:teuthology.orchestra.run.smithi080.stdout: # don't complain about insecure global_id in the test suite 2023-01-25T11:00:10.175 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-25T11:00:10.175 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-25T11:00:10.175 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.175 INFO:teuthology.orchestra.run.smithi080.stdout: # 1m isn't quite enough 2023-01-25T11:00:10.176 INFO:teuthology.orchestra.run.smithi080.stdout: mon_down_mkfs_grace = 2m 2023-01-25T11:00:10.176 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.176 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_filestore_osds = false 2023-01-25T11:00:10.176 INFO:teuthology.orchestra.run.smithi080.stdout: mon scrub interval = 300 2023-01-25T11:00:10.176 INFO:teuthology.orchestra.run.smithi080.stdout: 2023-01-25T11:00:10.177 INFO:teuthology.orchestra.run.smithi080.stdout:[client] 2023-01-25T11:00:10.177 INFO:teuthology.orchestra.run.smithi080.stdout: rgw cache enabled = true 2023-01-25T11:00:10.177 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable ops log = true 2023-01-25T11:00:10.177 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable usage log = true 2023-01-25T11:00:10.177 INFO:teuthology.orchestra.run.smithi080.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-25T11:00:10.178 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-25T11:00:10.178 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.a] 2023-01-25T11:00:10.178 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.c] 2023-01-25T11:00:10.178 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.b] 2023-01-25T11:00:10.178 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2023-01-25T11:00:10.178 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2023-01-25T11:00:10.179 INFO:teuthology.orchestra.run.smithi080.stdout: debug objecter = 20 2023-01-25T11:00:10.179 INFO:teuthology.orchestra.run.smithi080.stdout: debug rados = 20 2023-01-25T11:00:10.179 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-25T11:00:10.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-25T11:00:10.242 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-25T11:00:10.243 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:10.243 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-25T11:00:10.261 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:10.261 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-25T11:00:10.320 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T11:00:10.321 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:10.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-25T11:00:10.321 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-25T11:00:10.398 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:10.399 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:10.457 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi080.front.sepia.ceph.com 2023-01-25T11:00:10.458 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:10.458 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-25T11:00:10.459 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-25T11:00:10.497 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:10.497 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:10.555 INFO:tasks.ceph:Setting up mon nodes... 2023-01-25T11:00:10.556 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-25T11:00:10.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-01-25T11:00:10.611 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-01-25T11:00:10.613 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-25T11:00:10.691 INFO:teuthology.orchestra.run.smithi080.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-25T11:00:10.693 INFO:tasks.ceph:Setting up mds nodes... 2023-01-25T11:00:10.694 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-25T11:00:10.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-25T11:00:10.740 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-25T11:00:10.753 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-01-25T11:00:10.796 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-01-25T11:00:10.809 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-25T11:00:10.809 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-01-25T11:00:10.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-25T11:00:10.837 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:10.837 INFO:tasks.ceph:role: osd.0 2023-01-25T11:00:10.838 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T11:00:10.838 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-25T11:00:10.912 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:10.912 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:10.914 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:10.914 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:10.914 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:11.343 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-01-25T11:00:11.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-01-25T11:00:11.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-25T11:00:11.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-25T11:00:11.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-25T11:00:11.607 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:11.607 INFO:tasks.ceph:role: osd.1 2023-01-25T11:00:11.607 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T11:00:11.608 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-25T11:00:11.683 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:11.684 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:11.685 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:11.685 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:11.685 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:11.685 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:11.685 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:11.686 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:12.097 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-01-25T11:00:12.099 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-01-25T11:00:12.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-25T11:00:12.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-25T11:00:12.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-25T11:00:12.306 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:12.306 INFO:tasks.ceph:role: osd.2 2023-01-25T11:00:12.306 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T11:00:12.307 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-25T11:00:12.380 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:12.381 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:12.381 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:12.381 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:12.381 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:12.382 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:12.382 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:12.382 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:12.382 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:12.382 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:12.825 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-01-25T11:00:12.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-01-25T11:00:12.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-25T11:00:12.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-25T11:00:13.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-01-25T11:00:13.087 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:13.088 INFO:tasks.ceph:role: osd.3 2023-01-25T11:00:13.088 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T11:00:13.088 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-25T11:00:13.163 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:13.163 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:13.163 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:13.164 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:13.164 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:13.164 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:13.164 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:13.164 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:13.165 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:13.165 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:13.598 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-01-25T11:00:13.600 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-01-25T11:00:13.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-01-25T11:00:13.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-01-25T11:00:13.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:13.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:13.889+0000 7f5c87bed580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-25T11:00:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:13.889+0000 7f5c87bed580 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-25T11:00:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:13.889+0000 7f5c87bed580 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-25T11:00:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:13.889+0000 7f5c87bed580 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-25T11:00:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:13.889+0000 7f5c87bed580 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-25T11:00:13.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:13.890+0000 7f5c87bed580 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-25T11:00:14.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:14.414+0000 7f5c87bed580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:15.180+0000 7f5c87bed580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:15.950 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:15.947+0000 7f5c87bed580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:16.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-25T11:00:16.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:16.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:16.868+0000 7fce88984580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-25T11:00:16.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:16.868+0000 7fce88984580 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-25T11:00:16.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:16.868+0000 7fce88984580 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-25T11:00:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:16.868+0000 7fce88984580 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-25T11:00:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:16.868+0000 7fce88984580 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-25T11:00:16.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:16.868+0000 7fce88984580 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-25T11:00:17.398 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:17.395+0000 7fce88984580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:18.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:18.157+0000 7fce88984580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:18.931+0000 7fce88984580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:19.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-25T11:00:19.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:19.851 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:19.849+0000 7f0e31ae5580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-25T11:00:19.851 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:19.849+0000 7f0e31ae5580 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-25T11:00:19.851 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:19.849+0000 7f0e31ae5580 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-25T11:00:19.851 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:19.849+0000 7f0e31ae5580 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-25T11:00:19.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:19.849+0000 7f0e31ae5580 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-25T11:00:19.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:19.850+0000 7f0e31ae5580 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-25T11:00:20.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:20.379+0000 7f0e31ae5580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:21.143+0000 7f0e31ae5580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:21.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:21.906+0000 7f0e31ae5580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:22.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-25T11:00:22.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:22.824+0000 7f92565b0580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-01-25T11:00:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:22.824+0000 7f92565b0580 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-01-25T11:00:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:22.824+0000 7f92565b0580 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-25T11:00:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:22.824+0000 7f92565b0580 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-25T11:00:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:22.824+0000 7f92565b0580 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-25T11:00:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:22.824+0000 7f92565b0580 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-01-25T11:00:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:23.353+0000 7f92565b0580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:24.124+0000 7f92565b0580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:24.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-01-25T11:00:24.884+0000 7f92565b0580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:25.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-01-25T11:00:25.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-01-25T11:00:25.749 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:25.750 INFO:tasks.ceph:role: osd.4 2023-01-25T11:00:25.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi080.front.sepia.ceph.com 2023-01-25T11:00:25.750 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-25T11:00:25.828 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:25.828 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:25.829 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:25.829 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:25.829 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:25.830 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:25.830 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:25.830 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:26.263 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2023-01-25T11:00:26.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2023-01-25T11:00:26.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-01-25T11:00:26.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-01-25T11:00:26.478 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-01-25T11:00:26.546 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:26.547 INFO:tasks.ceph:role: osd.5 2023-01-25T11:00:26.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi080.front.sepia.ceph.com 2023-01-25T11:00:26.547 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-25T11:00:26.622 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:26.622 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:26.622 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:26.623 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:26.623 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:26.623 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:26.623 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:26.623 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:26.624 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:26.624 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:27.044 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2023-01-25T11:00:27.046 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2023-01-25T11:00:27.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-01-25T11:00:27.148 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-01-25T11:00:27.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-01-25T11:00:27.247 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:27.247 INFO:tasks.ceph:role: osd.6 2023-01-25T11:00:27.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi080.front.sepia.ceph.com 2023-01-25T11:00:27.248 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-25T11:00:27.321 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:27.321 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:27.322 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:27.322 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:27.323 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:27.323 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:27.323 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:27.323 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:27.768 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2023-01-25T11:00:27.770 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2023-01-25T11:00:27.770 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-01-25T11:00:27.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-01-25T11:00:27.956 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-01-25T11:00:28.024 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-25T11:00:28.025 INFO:tasks.ceph:role: osd.7 2023-01-25T11:00:28.025 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi080.front.sepia.ceph.com 2023-01-25T11:00:28.028 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-25T11:00:28.100 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T11:00:28.101 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T11:00:28.101 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T11:00:28.101 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T11:00:28.101 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T11:00:28.101 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2023-01-25T11:00:28.102 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T11:00:28.102 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T11:00:28.102 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T11:00:28.102 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T11:00:28.543 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2023-01-25T11:00:28.544 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2023-01-25T11:00:28.545 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-01-25T11:00:28.700 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-01-25T11:00:28.734 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:28.835 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:28.831+0000 7f2d3474c580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-01-25T11:00:28.835 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:28.831+0000 7f2d3474c580 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-01-25T11:00:28.835 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:28.831+0000 7f2d3474c580 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-25T11:00:28.836 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:28.831+0000 7f2d3474c580 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-25T11:00:28.836 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:28.831+0000 7f2d3474c580 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-25T11:00:28.836 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:28.832+0000 7f2d3474c580 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-01-25T11:00:29.359 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:29.355+0000 7f2d3474c580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:30.127 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:30.123+0000 7f2d3474c580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:30.890 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:30.887+0000 7f2d3474c580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:31.685 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-01-25T11:00:31.719 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:31.815 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:31.811+0000 7f6ea4592580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-01-25T11:00:31.815 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:31.812+0000 7f6ea4592580 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-01-25T11:00:31.816 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:31.812+0000 7f6ea4592580 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-25T11:00:31.816 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:31.812+0000 7f6ea4592580 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-25T11:00:31.816 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:31.812+0000 7f6ea4592580 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-25T11:00:31.816 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:31.812+0000 7f6ea4592580 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-01-25T11:00:32.339 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:32.336+0000 7f6ea4592580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:33.102 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:33.099+0000 7f6ea4592580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:33.866 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:33.863+0000 7f6ea4592580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:34.658 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-01-25T11:00:34.694 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:34.792 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:34.789+0000 7f6ae2703580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-01-25T11:00:34.792 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:34.789+0000 7f6ae2703580 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-01-25T11:00:34.793 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:34.789+0000 7f6ae2703580 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-25T11:00:34.793 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:34.789+0000 7f6ae2703580 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-25T11:00:34.793 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:34.789+0000 7f6ae2703580 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-25T11:00:34.794 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:34.790+0000 7f6ae2703580 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-01-25T11:00:35.319 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:35.317+0000 7f6ae2703580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:36.086 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:36.084+0000 7f6ae2703580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:36.851 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:36.848+0000 7f6ae2703580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:37.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-01-25T11:00:37.679 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:37.776 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:37.772+0000 7f05b7a63580 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-01-25T11:00:37.776 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:37.773+0000 7f05b7a63580 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-01-25T11:00:37.776 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:37.773+0000 7f05b7a63580 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-25T11:00:37.776 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:37.773+0000 7f05b7a63580 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-25T11:00:37.777 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:37.773+0000 7f05b7a63580 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-25T11:00:37.777 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:37.773+0000 7f05b7a63580 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-01-25T11:00:38.299 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:38.297+0000 7f05b7a63580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:39.060 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:39.057+0000 7f05b7a63580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:39.822 INFO:teuthology.orchestra.run.smithi080.stderr:2023-01-25T11:00:39.820+0000 7f05b7a63580 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T11:00:40.620 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-01-25T11:00:40.649 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-25T11:00:40.649 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:40.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-01-25T11:00:40.680 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:40.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-25T11:00:40.749 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:40.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-25T11:00:40.817 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:40.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-25T11:00:40.885 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:40.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-01-25T11:00:40.954 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:40.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-25T11:00:40.982 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:40.983 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-01-25T11:00:41.049 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:41.049 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-01-25T11:00:41.116 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:41.116 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-01-25T11:00:41.182 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:41.182 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-01-25T11:00:41.249 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:41.249 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-25T11:00:41.268 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:41.268 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-01-25T11:00:41.307 INFO:tasks.ceph:Adding keys to all mons... 2023-01-25T11:00:41.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-25T11:00:41.310 DEBUG:teuthology.orchestra.run.smithi080:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-25T11:00:41.350 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2023-01-25T11:00:41.350 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC6C9Fj3BJeJBAAnpZtr4tOEBpN7BlaBl/lTQ== 2023-01-25T11:00:41.350 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2023-01-25T11:00:41.351 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC9C9FjiXwXNRAAbajmBtnPIqoD3bQt/hpgCw== 2023-01-25T11:00:41.351 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2023-01-25T11:00:41.351 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDAC9FjwxDQMxAAUroveVpiVDDiu5OFPiX+Ig== 2023-01-25T11:00:41.351 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2023-01-25T11:00:41.351 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDDC9Fj8LKgMhAASUC+8TMM/u+LfCSOJa+QKg== 2023-01-25T11:00:41.352 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2023-01-25T11:00:41.352 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDGC9FjYCUiMRAAW0enCBC3BFd0267MJUSzng== 2023-01-25T11:00:41.352 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2023-01-25T11:00:41.352 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC6C9Fjp8cdKRAAwasHPX158hAgmLSD77K2ww== 2023-01-25T11:00:41.352 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2023-01-25T11:00:41.353 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDMC9Fj3YWoMRAA8EfS4ogrBo0W5lhCT/u3tA== 2023-01-25T11:00:41.353 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2023-01-25T11:00:41.353 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDPC9FjWveAMBAAupCkrhQ0135axdLFEoA9iQ== 2023-01-25T11:00:41.353 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2023-01-25T11:00:41.353 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDSC9FjGo4dLxAAdAIo3Fss8cOA8i31WN1vHA== 2023-01-25T11:00:41.354 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2023-01-25T11:00:41.354 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDVC9FjXicbLhAAV18BXBHWgCRga0UZd/n85A== 2023-01-25T11:00:41.354 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2023-01-25T11:00:41.354 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC6C9FjMvEMLBAAt0hqn8CCx7A+IBxVUrhrqA== 2023-01-25T11:00:41.354 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2023-01-25T11:00:41.355 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC6C9Fjp2BrLxAAnXJ3Mbd3KHYu6mxNYGl4Uw== 2023-01-25T11:00:41.372 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr.y] 2023-01-25T11:00:41.373 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQC6C9Fj3BJeJBAAnpZtr4tOEBpN7BlaBl/lTQ== 2023-01-25T11:00:41.373 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.0] 2023-01-25T11:00:41.373 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQC9C9FjiXwXNRAAbajmBtnPIqoD3bQt/hpgCw== 2023-01-25T11:00:41.373 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.1] 2023-01-25T11:00:41.374 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDAC9FjwxDQMxAAUroveVpiVDDiu5OFPiX+Ig== 2023-01-25T11:00:41.374 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.2] 2023-01-25T11:00:41.374 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDDC9Fj8LKgMhAASUC+8TMM/u+LfCSOJa+QKg== 2023-01-25T11:00:41.374 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.3] 2023-01-25T11:00:41.374 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDGC9FjYCUiMRAAW0enCBC3BFd0267MJUSzng== 2023-01-25T11:00:41.375 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr.x] 2023-01-25T11:00:41.375 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQC6C9Fjp8cdKRAAwasHPX158hAgmLSD77K2ww== 2023-01-25T11:00:41.375 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.4] 2023-01-25T11:00:41.375 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDMC9Fj3YWoMRAA8EfS4ogrBo0W5lhCT/u3tA== 2023-01-25T11:00:41.375 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.5] 2023-01-25T11:00:41.375 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDPC9FjWveAMBAAupCkrhQ0135axdLFEoA9iQ== 2023-01-25T11:00:41.376 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.6] 2023-01-25T11:00:41.376 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDSC9FjGo4dLxAAdAIo3Fss8cOA8i31WN1vHA== 2023-01-25T11:00:41.376 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.7] 2023-01-25T11:00:41.376 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQDVC9FjXicbLhAAV18BXBHWgCRga0UZd/n85A== 2023-01-25T11:00:41.376 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2023-01-25T11:00:41.377 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQC6C9FjMvEMLBAAt0hqn8CCx7A+IBxVUrhrqA== 2023-01-25T11:00:41.377 INFO:teuthology.orchestra.run.smithi080.stdout:[client.1] 2023-01-25T11:00:41.377 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQC6C9Fjp2BrLxAAnXJ3Mbd3KHYu6mxNYGl4Uw== 2023-01-25T11:00:41.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-25T11:00:41.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-25T11:00:41.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.473 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.529 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.584 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.640 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-25T11:00:41.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-25T11:00:41.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.751 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.807 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.863 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.919 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T11:00:41.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-25T11:00:41.975 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-25T11:00:42.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-25T11:00:42.034 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-25T11:00:42.091 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-25T11:00:42.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-25T11:00:42.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-25T11:00:42.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-25T11:00:42.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-01-25T11:00:42.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-25T11:00:42.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-01-25T11:00:42.445 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-01-25T11:00:42.473 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-25T11:00:42.573 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-01-25T11:00:42.600 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:42.602 DEBUG:teuthology.orchestra.run.smithi080:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-25T11:00:42.658 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-25T11:00:42.658 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-25T11:00:42.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-25T11:00:42.661 INFO:tasks.ceph.mon.a:Started 2023-01-25T11:00:42.661 INFO:tasks.ceph.mon.c:Restarting daemon 2023-01-25T11:00:42.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-01-25T11:00:42.664 INFO:tasks.ceph.mon.c:Started 2023-01-25T11:00:42.664 INFO:tasks.ceph.mon.b:Restarting daemon 2023-01-25T11:00:42.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-01-25T11:00:42.700 INFO:tasks.ceph.mon.b:Started 2023-01-25T11:00:42.701 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-25T11:00:42.701 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-01-25T11:00:42.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-01-25T11:00:42.704 INFO:tasks.ceph.mgr.y:Started 2023-01-25T11:00:42.705 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-25T11:00:42.705 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-25T11:00:42.707 INFO:tasks.ceph.mgr.x:Started 2023-01-25T11:00:42.707 INFO:tasks.ceph:Setting crush tunables to default 2023-01-25T11:00:42.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2023-01-25T11:00:43.099 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2023-01-25T11:00:43.114 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-25T11:00:43.115 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:43.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-25T11:00:43.143 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:43.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-25T11:00:43.211 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:43.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-25T11:00:43.279 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-01-25T11:00:43.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-01-25T11:00:43.347 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:43.347 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-01-25T11:00:43.375 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:43.375 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-01-25T11:00:43.442 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:43.443 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-01-25T11:00:43.511 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2023-01-25T11:00:43.511 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-01-25T11:00:43.578 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new 480195f5-49b9-4456-bf14-fffd5c2fc726 0 2023-01-25T11:00:43.838 INFO:teuthology.orchestra.run.smithi080.stdout:0 2023-01-25T11:00:43.851 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new fa593ec2-e991-4229-aa40-e47a62062d0b 1 2023-01-25T11:00:43.958 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:43.955+0000 7f28721e3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-25T11:00:44.032 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:44.029+0000 7f28721e3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-25T11:00:44.068 INFO:teuthology.orchestra.run.smithi080.stdout:1 2023-01-25T11:00:44.080 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new a82bac40-3ce2-4590-8c25-f11e17380406 2 2023-01-25T11:00:44.122 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:44.120+0000 7f28721e3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-25T11:00:44.299 INFO:teuthology.orchestra.run.smithi080.stdout:2 2023-01-25T11:00:44.312 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new 3a11d64b-b3b0-4fe8-b0e5-620841995b30 3 2023-01-25T11:00:44.316 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:44.314+0000 7f28721e3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-25T11:00:44.414 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:44.412+0000 7f1a4d130200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-25T11:00:44.487 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:44.485+0000 7f1a4d130200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-25T11:00:44.534 INFO:teuthology.orchestra.run.smithi080.stdout:3 2023-01-25T11:00:44.547 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new 2185db30-08a8-44f8-8068-ce3924f9cdc2 4 2023-01-25T11:00:44.585 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:44.583+0000 7f1a4d130200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-25T11:00:44.765 INFO:teuthology.orchestra.run.smithi080.stdout:4 2023-01-25T11:00:44.777 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:44.775+0000 7f1a4d130200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-25T11:00:44.778 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new 36f79b7c-7efe-4a70-9d1e-f3bf1bce014d 5 2023-01-25T11:00:45.035 INFO:teuthology.orchestra.run.smithi080.stdout:5 2023-01-25T11:00:45.047 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new 52b51a13-0466-4d84-8d89-0d745742af3c 6 2023-01-25T11:00:45.158 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:45.155+0000 7f28721e3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-25T11:00:45.263 INFO:teuthology.orchestra.run.smithi080.stdout:6 2023-01-25T11:00:45.276 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster ceph osd new cca39342-3b39-4c30-af7c-255eb6fe821c 7 2023-01-25T11:00:45.305 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:45.302+0000 7f28721e3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-25T11:00:45.525 INFO:teuthology.orchestra.run.smithi080.stdout:7 2023-01-25T11:00:45.538 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-25T11:00:45.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-25T11:00:45.539 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:45.536+0000 7f28721e3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-25T11:00:45.541 INFO:tasks.ceph.osd.0:Started 2023-01-25T11:00:45.541 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-25T11:00:45.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-25T11:00:45.543 INFO:tasks.ceph.osd.1:Started 2023-01-25T11:00:45.544 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-25T11:00:45.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-25T11:00:45.546 INFO:tasks.ceph.osd.2:Started 2023-01-25T11:00:45.546 INFO:tasks.ceph.osd.3:Restarting daemon 2023-01-25T11:00:45.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-01-25T11:00:45.549 INFO:tasks.ceph.osd.3:Started 2023-01-25T11:00:45.549 INFO:tasks.ceph.osd.4:Restarting daemon 2023-01-25T11:00:45.549 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-01-25T11:00:45.551 INFO:tasks.ceph.osd.4:Started 2023-01-25T11:00:45.552 INFO:tasks.ceph.osd.5:Restarting daemon 2023-01-25T11:00:45.552 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-01-25T11:00:45.554 INFO:tasks.ceph.osd.5:Started 2023-01-25T11:00:45.554 INFO:tasks.ceph.osd.6:Restarting daemon 2023-01-25T11:00:45.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-01-25T11:00:45.557 INFO:tasks.ceph.osd.6:Started 2023-01-25T11:00:45.557 INFO:tasks.ceph.osd.7:Restarting daemon 2023-01-25T11:00:45.557 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-01-25T11:00:45.559 INFO:tasks.ceph.osd.7:Started 2023-01-25T11:00:45.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T11:00:45.611 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:45.609+0000 7f1a4d130200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-25T11:00:45.644 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:45.642+0000 7f28721e3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-25T11:00:45.760 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:45.758+0000 7f1a4d130200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-25T11:00:45.794 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:45.791+0000 7f28721e3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-25T11:00:45.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:45.795 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"fsid":"be11100e-61e6-4b91-8166-d134ac9fbd8b","created":"2023-01-25T11:00:42.977818+0000","modified":"2023-01-25T11:00:45.521234+0000","last_up_change":"0.000000","last_in_change":"2023-01-25T11:00:45.521234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"480195f5-49b9-4456-bf14-fffd5c2fc726","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fa593ec2-e991-4229-aa40-e47a62062d0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a82bac40-3ce2-4590-8c25-f11e17380406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3a11d64b-b3b0-4fe8-b0e5-620841995b30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2185db30-08a8-44f8-8068-ce3924f9cdc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36f79b7c-7efe-4a70-9d1e-f3bf1bce014d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"52b51a13-0466-4d84-8d89-0d745742af3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cca39342-3b39-4c30-af7c-255eb6fe821c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T11:00:45.808 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-25T11:00:45.808 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-25T11:00:45.981 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:45.979+0000 7f1a4d130200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-25T11:00:46.053 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:46.050+0000 7f1a4d130200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-25T11:00:46.079 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.076+0000 7f28721e3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-25T11:00:46.159 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:00:46.155+0000 7ffaf1a41580 -1 Falling back to public interface 2023-01-25T11:00:46.167 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:00:46.164+0000 7f5f4ecd2580 -1 Falling back to public interface 2023-01-25T11:00:46.169 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:00:46.167+0000 7fb8bc072580 -1 Falling back to public interface 2023-01-25T11:00:46.172 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:00:46.169+0000 7fac99232580 -1 Falling back to public interface 2023-01-25T11:00:46.173 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:00:46.169+0000 7fc3e2bdf580 -1 Falling back to public interface 2023-01-25T11:00:46.173 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:00:46.170+0000 7fc3d9f8d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2023-01-25T11:00:46.174 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:00:46.171+0000 7fa9a122b580 -1 Falling back to public interface 2023-01-25T11:00:46.183 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:00:46.181+0000 7f4d4890f580 -1 Falling back to public interface 2023-01-25T11:00:46.185 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:00:46.182+0000 7fcc4c619580 -1 Falling back to public interface 2023-01-25T11:00:46.190 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:46.188+0000 7f1a4d130200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-25T11:00:46.272 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.269+0000 7f28721e3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-25T11:00:46.418 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.415+0000 7f28721e3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-25T11:00:46.454 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:46.452+0000 7f1a4d130200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-25T11:00:46.543 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.536+0000 7f28721e3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-25T11:00:46.614 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-25T11:00:46.637 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:46.634+0000 7f1a4d130200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-25T11:00:46.685 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.682+0000 7f28721e3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-25T11:00:46.772 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.769+0000 7f28721e3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-25T11:00:46.776 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:46.774+0000 7f1a4d130200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-25T11:00:46.811 INFO:teuthology.misc.health.smithi027.stdout: 2023-01-25T11:00:46.812 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"be11100e-61e6-4b91-8166-d134ac9fbd8b","created":"2023-01-25T11:00:42.977818+0000","modified":"2023-01-25T11:00:45.521234+0000","last_up_change":"0.000000","last_in_change":"2023-01-25T11:00:45.521234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"480195f5-49b9-4456-bf14-fffd5c2fc726","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fa593ec2-e991-4229-aa40-e47a62062d0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a82bac40-3ce2-4590-8c25-f11e17380406","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3a11d64b-b3b0-4fe8-b0e5-620841995b30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2185db30-08a8-44f8-8068-ce3924f9cdc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"36f79b7c-7efe-4a70-9d1e-f3bf1bce014d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"52b51a13-0466-4d84-8d89-0d745742af3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cca39342-3b39-4c30-af7c-255eb6fe821c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T11:00:46.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-01-25T11:00:46.889 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:46.887+0000 7f1a4d130200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-25T11:00:47.000 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:46.998+0000 7f28721e3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-25T11:00:47.029 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:47.027+0000 7f1a4d130200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-25T11:00:47.081 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:47.079+0000 7f28721e3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-25T11:00:47.106 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:47.104+0000 7f1a4d130200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-25T11:00:47.325 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:47.322+0000 7f1a4d130200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-25T11:00:47.401 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:47.399+0000 7f1a4d130200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-25T11:00:47.735 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:47.733+0000 7f28721e3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-25T11:00:48.001 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:47.998+0000 7f28721e3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-25T11:00:48.014 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.012+0000 7f1a4d130200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-25T11:00:48.079 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:48.076+0000 7f28721e3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-25T11:00:48.165 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:48.162+0000 7f28721e3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-25T11:00:48.269 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.266+0000 7f1a4d130200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-25T11:00:48.343 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.341+0000 7f1a4d130200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-25T11:00:48.356 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:48.354+0000 7f28721e3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-25T11:00:48.425 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.422+0000 7f1a4d130200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-25T11:00:48.432 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:48.429+0000 7f28721e3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-25T11:00:48.590 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:48.587+0000 7f28721e3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-25T11:00:48.609 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.607+0000 7f1a4d130200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-25T11:00:48.684 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.682+0000 7f1a4d130200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-25T11:00:48.833 INFO:tasks.ceph.mgr.x.smithi080.stderr:2023-01-25T11:00:48.831+0000 7f1a4d130200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-25T11:00:49.615 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:49.613+0000 7f282bd5c700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-01-25T11:00:49.616 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-01-25T11:00:49.613+0000 7f282bd5c700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-01-25T11:00:49.632 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:00:49.629+0000 7ffaf1a41580 -1 osd.4 0 log_to_monitors true 2023-01-25T11:00:49.651 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:00:49.649+0000 7fb8bc072580 -1 osd.7 0 log_to_monitors true 2023-01-25T11:00:49.660 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:00:49.658+0000 7fc3e2bdf580 -1 osd.2 0 log_to_monitors true 2023-01-25T11:00:49.670 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:00:49.668+0000 7fa9a122b580 -1 osd.5 0 log_to_monitors true 2023-01-25T11:00:49.680 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:00:49.677+0000 7f5f4ecd2580 -1 osd.0 0 log_to_monitors true 2023-01-25T11:00:49.687 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:00:49.684+0000 7f4d4890f580 -1 osd.6 0 log_to_monitors true 2023-01-25T11:00:49.689 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:00:49.687+0000 7fcc4c619580 -1 osd.3 0 log_to_monitors true 2023-01-25T11:00:49.698 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:00:49.695+0000 7fac99232580 -1 osd.1 0 log_to_monitors true 2023-01-25T11:00:51.620 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:00:51.618+0000 7ffae55e8700 -1 osd.4 0 waiting for initial osdmap 2023-01-25T11:00:51.621 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:00:51.618+0000 7fb8afc19700 -1 osd.7 0 waiting for initial osdmap 2023-01-25T11:00:51.621 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:00:51.618+0000 7ffad8310700 -1 osd.4 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.622 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:00:51.618+0000 7fb8a2941700 -1 osd.7 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.622 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:00:51.619+0000 7fa994dd2700 -1 osd.5 0 waiting for initial osdmap 2023-01-25T11:00:51.623 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:00:51.619+0000 7fa987afa700 -1 osd.5 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.623 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:00:51.619+0000 7fc3d6786700 -1 osd.2 0 waiting for initial osdmap 2023-01-25T11:00:51.624 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:00:51.619+0000 7fc3c94ae700 -1 osd.2 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.624 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:00:51.621+0000 7fcc401c0700 -1 osd.3 0 waiting for initial osdmap 2023-01-25T11:00:51.624 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:00:51.621+0000 7fcc32ee8700 -1 osd.3 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.629 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:00:51.627+0000 7f4d3c4b6700 -1 osd.6 0 waiting for initial osdmap 2023-01-25T11:00:51.630 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:00:51.627+0000 7f5f42879700 -1 osd.0 0 waiting for initial osdmap 2023-01-25T11:00:51.630 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:00:51.627+0000 7f5f355a1700 -1 osd.0 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.631 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:00:51.629+0000 7f4d2f1de700 -1 osd.6 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:00:51.629+0000 7fac8cdd9700 -1 osd.1 0 waiting for initial osdmap 2023-01-25T11:00:51.632 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:00:51.629+0000 7fac7fb01700 -1 osd.1 0 handle_osd_map injecting map crc failure 2023-01-25T11:00:51.632 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:00:51.630+0000 7ffadd9c3700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.633 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:00:51.631+0000 7fa98d1ad700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.635 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:00:51.633+0000 7f5f3ac54700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.636 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:00:51.633+0000 7fb8a7ff4700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.636 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:00:51.633+0000 7fc3ceb61700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.636 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:00:51.633+0000 7fac851b4700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.637 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:00:51.633+0000 7fcc3859b700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:51.637 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:00:51.634+0000 7f4d34891700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T11:00:53.630 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-25T11:00:53.964 INFO:teuthology.misc.health.smithi027.stdout: 2023-01-25T11:00:53.964 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":14,"fsid":"be11100e-61e6-4b91-8166-d134ac9fbd8b","created":"2023-01-25T11:00:42.977818+0000","modified":"2023-01-25T11:00:53.636416+0000","last_up_change":"2023-01-25T11:00:52.623244+0000","last_in_change":"2023-01-25T11:00:45.521234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-25T11:00:52.671604+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"480195f5-49b9-4456-bf14-fffd5c2fc726","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":106419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":106419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":106419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":106419}]},"public_addr":"172.21.15.27:6800/106419","cluster_addr":"172.21.15.27:6801/106419","heartbeat_back_addr":"172.21.15.27:6803/106419","heartbeat_front_addr":"172.21.15.27:6802/106419","state":["exists","up"]},{"osd":1,"uuid":"fa593ec2-e991-4229-aa40-e47a62062d0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":106422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":106422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":106422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":106422}]},"public_addr":"172.21.15.27:6804/106422","cluster_addr":"172.21.15.27:6807/106422","heartbeat_back_addr":"172.21.15.27:6811/106422","heartbeat_front_addr":"172.21.15.27:6809/106422","state":["exists","up"]},{"osd":2,"uuid":"a82bac40-3ce2-4590-8c25-f11e17380406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":106421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":106421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":106421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":106421}]},"public_addr":"172.21.15.27:6805/106421","cluster_addr":"172.21.15.27:6806/106421","heartbeat_back_addr":"172.21.15.27:6810/106421","heartbeat_front_addr":"172.21.15.27:6808/106421","state":["exists","up"]},{"osd":3,"uuid":"3a11d64b-b3b0-4fe8-b0e5-620841995b30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":106431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":106431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":106431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":106431}]},"public_addr":"172.21.15.27:6812/106431","cluster_addr":"172.21.15.27:6813/106431","heartbeat_back_addr":"172.21.15.27:6815/106431","heartbeat_front_addr":"172.21.15.27:6814/106431","state":["exists","up"]},{"osd":4,"uuid":"2185db30-08a8-44f8-8068-ce3924f9cdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":106320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6801","nonce":106320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6803","nonce":106320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":106320}]},"public_addr":"172.21.15.80:6800/106320","cluster_addr":"172.21.15.80:6801/106320","heartbeat_back_addr":"172.21.15.80:6803/106320","heartbeat_front_addr":"172.21.15.80:6802/106320","state":["exists","up"]},{"osd":5,"uuid":"36f79b7c-7efe-4a70-9d1e-f3bf1bce014d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":106330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6809","nonce":106330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6811","nonce":106330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":106330}]},"public_addr":"172.21.15.80:6808/106330","cluster_addr":"172.21.15.80:6809/106330","heartbeat_back_addr":"172.21.15.80:6811/106330","heartbeat_front_addr":"172.21.15.80:6810/106330","state":["exists","up"]},{"osd":6,"uuid":"52b51a13-0466-4d84-8d89-0d745742af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":106337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6813","nonce":106337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6815","nonce":106337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":106337}]},"public_addr":"172.21.15.80:6812/106337","cluster_addr":"172.21.15.80:6813/106337","heartbeat_back_addr":"172.21.15.80:6815/106337","heartbeat_front_addr":"172.21.15.80:6814/106337","state":["exists","up"]},{"osd":7,"uuid":"cca39342-3b39-4c30-af7c-255eb6fe821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":106321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6805","nonce":106321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6807","nonce":106321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":106321}]},"public_addr":"172.21.15.80:6804/106321","cluster_addr":"172.21.15.80:6805/106321","heartbeat_back_addr":"172.21.15.80:6807/106321","heartbeat_front_addr":"172.21.15.80:6806/106321","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.706841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.747792+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.649187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.690417+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.596143+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.622733+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.681465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.664684+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T11:00:53.977 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-01-25T11:00:53.977 INFO:tasks.ceph:Creating RBD pool 2023-01-25T11:00:53.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-25T11:00:54.654 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:00:54.652+0000 7fb8a2941700 -1 osd.7 14 handle_osd_map injecting map crc failure 2023-01-25T11:00:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2023-01-25T11:00:54.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-25T11:00:55.664 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-25T11:00:55.678 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-25T11:00:55.679 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-25T11:00:55.679 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-25T11:00:55.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-25T11:00:55.680 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-25T11:00:56.071 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:56.084 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":4,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":106122}]},"active_addr":"172.21.15.27:6816/106122","active_change":"2023-01-25T11:00:48.602677+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4111,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1175803630}]},{"addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2439073985}]},{"addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2328267922}]}]}} 2023-01-25T11:00:56.088 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-25T11:00:56.088 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-25T11:00:56.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T11:00:56.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:56.419 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"fsid":"be11100e-61e6-4b91-8166-d134ac9fbd8b","created":"2023-01-25T11:00:42.977818+0000","modified":"2023-01-25T11:00:55.646991+0000","last_up_change":"2023-01-25T11:00:52.623244+0000","last_in_change":"2023-01-25T11:00:45.521234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-25T11:00:52.671604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-25T11:00:54.335175+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"480195f5-49b9-4456-bf14-fffd5c2fc726","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":106419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":106419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":106419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":106419}]},"public_addr":"172.21.15.27:6800/106419","cluster_addr":"172.21.15.27:6801/106419","heartbeat_back_addr":"172.21.15.27:6803/106419","heartbeat_front_addr":"172.21.15.27:6802/106419","state":["exists","up"]},{"osd":1,"uuid":"fa593ec2-e991-4229-aa40-e47a62062d0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":106422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":106422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":106422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":106422}]},"public_addr":"172.21.15.27:6804/106422","cluster_addr":"172.21.15.27:6807/106422","heartbeat_back_addr":"172.21.15.27:6811/106422","heartbeat_front_addr":"172.21.15.27:6809/106422","state":["exists","up"]},{"osd":2,"uuid":"a82bac40-3ce2-4590-8c25-f11e17380406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":106421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":106421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":106421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":106421}]},"public_addr":"172.21.15.27:6805/106421","cluster_addr":"172.21.15.27:6806/106421","heartbeat_back_addr":"172.21.15.27:6810/106421","heartbeat_front_addr":"172.21.15.27:6808/106421","state":["exists","up"]},{"osd":3,"uuid":"3a11d64b-b3b0-4fe8-b0e5-620841995b30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":106431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":106431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":106431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":106431}]},"public_addr":"172.21.15.27:6812/106431","cluster_addr":"172.21.15.27:6813/106431","heartbeat_back_addr":"172.21.15.27:6815/106431","heartbeat_front_addr":"172.21.15.27:6814/106431","state":["exists","up"]},{"osd":4,"uuid":"2185db30-08a8-44f8-8068-ce3924f9cdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":106320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6801","nonce":106320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6803","nonce":106320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":106320}]},"public_addr":"172.21.15.80:6800/106320","cluster_addr":"172.21.15.80:6801/106320","heartbeat_back_addr":"172.21.15.80:6803/106320","heartbeat_front_addr":"172.21.15.80:6802/106320","state":["exists","up"]},{"osd":5,"uuid":"36f79b7c-7efe-4a70-9d1e-f3bf1bce014d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":106330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6809","nonce":106330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6811","nonce":106330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":106330}]},"public_addr":"172.21.15.80:6808/106330","cluster_addr":"172.21.15.80:6809/106330","heartbeat_back_addr":"172.21.15.80:6811/106330","heartbeat_front_addr":"172.21.15.80:6810/106330","state":["exists","up"]},{"osd":6,"uuid":"52b51a13-0466-4d84-8d89-0d745742af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":106337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6813","nonce":106337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6815","nonce":106337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":106337}]},"public_addr":"172.21.15.80:6812/106337","cluster_addr":"172.21.15.80:6813/106337","heartbeat_back_addr":"172.21.15.80:6815/106337","heartbeat_front_addr":"172.21.15.80:6814/106337","state":["exists","up"]},{"osd":7,"uuid":"cca39342-3b39-4c30-af7c-255eb6fe821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":106321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6805","nonce":106321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6807","nonce":106321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":106321}]},"public_addr":"172.21.15.80:6804/106321","cluster_addr":"172.21.15.80:6805/106321","heartbeat_back_addr":"172.21.15.80:6807/106321","heartbeat_front_addr":"172.21.15.80:6806/106321","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.706841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.747792+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.649187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.690417+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.596143+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.622733+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.681465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.664684+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T11:00:56.433 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-25T11:00:56.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T11:00:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"be11100e-61e6-4b91-8166-d134ac9fbd8b","created":"2023-01-25T11:00:42.977818+0000","modified":"2023-01-25T11:00:56.659714+0000","last_up_change":"2023-01-25T11:00:52.623244+0000","last_in_change":"2023-01-25T11:00:45.521234+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-25T11:00:52.671604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-25T11:00:54.335175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"480195f5-49b9-4456-bf14-fffd5c2fc726","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":106419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":106419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":106419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":106419}]},"public_addr":"172.21.15.27:6800/106419","cluster_addr":"172.21.15.27:6801/106419","heartbeat_back_addr":"172.21.15.27:6803/106419","heartbeat_front_addr":"172.21.15.27:6802/106419","state":["exists","up"]},{"osd":1,"uuid":"fa593ec2-e991-4229-aa40-e47a62062d0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":106422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":106422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":106422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":106422}]},"public_addr":"172.21.15.27:6804/106422","cluster_addr":"172.21.15.27:6807/106422","heartbeat_back_addr":"172.21.15.27:6811/106422","heartbeat_front_addr":"172.21.15.27:6809/106422","state":["exists","up"]},{"osd":2,"uuid":"a82bac40-3ce2-4590-8c25-f11e17380406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":106421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":106421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":106421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":106421}]},"public_addr":"172.21.15.27:6805/106421","cluster_addr":"172.21.15.27:6806/106421","heartbeat_back_addr":"172.21.15.27:6810/106421","heartbeat_front_addr":"172.21.15.27:6808/106421","state":["exists","up"]},{"osd":3,"uuid":"3a11d64b-b3b0-4fe8-b0e5-620841995b30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":106431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":106431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":106431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":106431}]},"public_addr":"172.21.15.27:6812/106431","cluster_addr":"172.21.15.27:6813/106431","heartbeat_back_addr":"172.21.15.27:6815/106431","heartbeat_front_addr":"172.21.15.27:6814/106431","state":["exists","up"]},{"osd":4,"uuid":"2185db30-08a8-44f8-8068-ce3924f9cdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":106320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6801","nonce":106320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6803","nonce":106320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":106320}]},"public_addr":"172.21.15.80:6800/106320","cluster_addr":"172.21.15.80:6801/106320","heartbeat_back_addr":"172.21.15.80:6803/106320","heartbeat_front_addr":"172.21.15.80:6802/106320","state":["exists","up"]},{"osd":5,"uuid":"36f79b7c-7efe-4a70-9d1e-f3bf1bce014d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":106330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6809","nonce":106330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6811","nonce":106330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":106330}]},"public_addr":"172.21.15.80:6808/106330","cluster_addr":"172.21.15.80:6809/106330","heartbeat_back_addr":"172.21.15.80:6811/106330","heartbeat_front_addr":"172.21.15.80:6810/106330","state":["exists","up"]},{"osd":6,"uuid":"52b51a13-0466-4d84-8d89-0d745742af3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":106337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6813","nonce":106337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6815","nonce":106337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":106337}]},"public_addr":"172.21.15.80:6812/106337","cluster_addr":"172.21.15.80:6813/106337","heartbeat_back_addr":"172.21.15.80:6815/106337","heartbeat_front_addr":"172.21.15.80:6814/106337","state":["exists","up"]},{"osd":7,"uuid":"cca39342-3b39-4c30-af7c-255eb6fe821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":106321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6805","nonce":106321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6807","nonce":106321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":106321}]},"public_addr":"172.21.15.80:6804/106321","cluster_addr":"172.21.15.80:6805/106321","heartbeat_back_addr":"172.21.15.80:6807/106321","heartbeat_front_addr":"172.21.15.80:6806/106321","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.706841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.747792+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.649187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.690417+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.596143+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.622733+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.681465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T11:00:50.664684+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T11:00:56.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-25T11:00:56.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-25T11:00:56.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-25T11:00:56.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-01-25T11:00:56.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-01-25T11:00:56.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-01-25T11:00:56.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-01-25T11:00:56.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-01-25T11:00:57.079 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:57.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-25T11:00:57.084 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:57.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-25T11:00:57.095 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:57.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-25T11:00:57.097 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:57.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-25T11:00:57.099 INFO:teuthology.orchestra.run.smithi027.stdout:55834574851 2023-01-25T11:00:57.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-25T11:00:57.101 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:57.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-25T11:00:57.110 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:57.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-25T11:00:57.185 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:57.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-25T11:00:57.727 INFO:teuthology.orchestra.run.smithi027.stdout:55834574851 2023-01-25T11:00:57.728 INFO:teuthology.orchestra.run.smithi027.stdout:55834574849 2023-01-25T11:00:57.730 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:57.733 INFO:teuthology.orchestra.run.smithi027.stdout:55834574851 2023-01-25T11:00:57.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.4 2023-01-25T11:00:57.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.2 2023-01-25T11:00:57.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2023-01-25T11:00:57.765 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.1 2023-01-25T11:00:57.766 INFO:teuthology.orchestra.run.smithi027.stdout:55834574851 2023-01-25T11:00:57.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2023-01-25T11:00:57.790 INFO:teuthology.orchestra.run.smithi027.stdout:55834574849 2023-01-25T11:00:57.795 INFO:teuthology.orchestra.run.smithi027.stdout:55834574849 2023-01-25T11:00:57.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.0 2023-01-25T11:00:57.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.6 2023-01-25T11:00:57.827 INFO:teuthology.orchestra.run.smithi027.stdout:55834574851 2023-01-25T11:00:57.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2023-01-25T11:00:58.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-25T11:00:58.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-25T11:00:58.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-25T11:00:58.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-25T11:00:58.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-25T11:00:58.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-25T11:00:58.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-25T11:00:58.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-25T11:00:59.369 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:59.388 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.4 2023-01-25T11:00:59.389 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.401 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:59.414 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:59.424 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.2 2023-01-25T11:00:59.424 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.436 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2023-01-25T11:00:59.437 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.439 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:59.445 INFO:teuthology.orchestra.run.smithi027.stdout:55834574851 2023-01-25T11:00:59.451 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:59.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.0 2023-01-25T11:00:59.456 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.469 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.1 2023-01-25T11:00:59.469 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.472 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2023-01-25T11:00:59.472 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.496 INFO:teuthology.orchestra.run.smithi027.stdout:55834574852 2023-01-25T11:00:59.509 INFO:teuthology.orchestra.run.smithi027.stdout:55834574850 2023-01-25T11:00:59.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2023-01-25T11:00:59.511 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.6 2023-01-25T11:00:59.524 DEBUG:teuthology.parallel:result is None 2023-01-25T11:00:59.524 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-25T11:00:59.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T11:00:59.895 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:00:59.941 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-01-25T11:00:59.941 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-01-25T11:00:58.609832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":40544,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":754934176,"statfs":{"total":773094113280,"available":773052596224,"internally_reserved":0,"allocated":1015808,"data_stored":572160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.750869+0000","last_change":"2023-01-25T11:00:55.750203+0000","last_active":"2023-01-25T11:00:55.750869+0000","last_peered":"2023-01-25T11:00:55.750869+0000","last_clean":"2023-01-25T11:00:55.750869+0000","last_became_active":"2023-01-25T11:00:55.749892+0000","last_became_peered":"2023-01-25T11:00:55.749892+0000","last_unstale":"2023-01-25T11:00:55.750869+0000","last_undegraded":"2023-01-25T11:00:55.750869+0000","last_fullsized":"2023-01-25T11:00:55.750869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:09:11.429769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.672264+0000","last_change":"2023-01-25T11:00:55.662042+0000","last_active":"2023-01-25T11:00:56.672264+0000","last_peered":"2023-01-25T11:00:56.672264+0000","last_clean":"2023-01-25T11:00:56.672264+0000","last_became_active":"2023-01-25T11:00:55.661764+0000","last_became_peered":"2023-01-25T11:00:55.661764+0000","last_unstale":"2023-01-25T11:00:56.672264+0000","last_undegraded":"2023-01-25T11:00:56.672264+0000","last_fullsized":"2023-01-25T11:00:56.672264+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:04:49.567058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.670623+0000","last_change":"2023-01-25T11:00:55.666198+0000","last_active":"2023-01-25T11:00:56.670623+0000","last_peered":"2023-01-25T11:00:56.670623+0000","last_clean":"2023-01-25T11:00:56.670623+0000","last_became_active":"2023-01-25T11:00:55.665948+0000","last_became_peered":"2023-01-25T11:00:55.665948+0000","last_unstale":"2023-01-25T11:00:56.670623+0000","last_undegraded":"2023-01-25T11:00:56.670623+0000","last_fullsized":"2023-01-25T11:00:56.670623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:29:29.559334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.670623+0000","last_change":"2023-01-25T11:00:55.891233+0000","last_active":"2023-01-25T11:00:56.670623+0000","last_peered":"2023-01-25T11:00:56.670623+0000","last_clean":"2023-01-25T11:00:56.670623+0000","last_became_active":"2023-01-25T11:00:55.890849+0000","last_became_peered":"2023-01-25T11:00:55.890849+0000","last_unstale":"2023-01-25T11:00:56.670623+0000","last_undegraded":"2023-01-25T11:00:56.670623+0000","last_fullsized":"2023-01-25T11:00:56.670623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:22:26.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'34","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.933860+0000","last_change":"2023-01-25T11:00:54.663781+0000","last_active":"2023-01-25T11:00:56.933860+0000","last_peered":"2023-01-25T11:00:56.933860+0000","last_clean":"2023-01-25T11:00:56.933860+0000","last_became_active":"2023-01-25T11:00:54.663067+0000","last_became_peered":"2023-01-25T11:00:54.663067+0000","last_unstale":"2023-01-25T11:00:56.933860+0000","last_undegraded":"2023-01-25T11:00:56.933860+0000","last_fullsized":"2023-01-25T11:00:56.933860+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_clean_scrub_stamp":"2023-01-25T11:00:53.636416+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:09:10.422745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.660459+0000","last_change":"2023-01-25T11:00:55.659957+0000","last_active":"2023-01-25T11:00:55.660459+0000","last_peered":"2023-01-25T11:00:55.660459+0000","last_clean":"2023-01-25T11:00:55.660459+0000","last_became_active":"2023-01-25T11:00:55.659687+0000","last_became_peered":"2023-01-25T11:00:55.659687+0000","last_unstale":"2023-01-25T11:00:55.660459+0000","last_undegraded":"2023-01-25T11:00:55.660459+0000","last_fullsized":"2023-01-25T11:00:55.660459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T22:07:34.953392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.757970+0000","last_change":"2023-01-25T11:00:55.757378+0000","last_active":"2023-01-25T11:00:55.757970+0000","last_peered":"2023-01-25T11:00:55.757970+0000","last_clean":"2023-01-25T11:00:55.757970+0000","last_became_active":"2023-01-25T11:00:55.757013+0000","last_became_peered":"2023-01-25T11:00:55.757013+0000","last_unstale":"2023-01-25T11:00:55.757970+0000","last_undegraded":"2023-01-25T11:00:55.757970+0000","last_fullsized":"2023-01-25T11:00:55.757970+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:58:57.108698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.663913+0000","last_change":"2023-01-25T11:00:55.662604+0000","last_active":"2023-01-25T11:00:55.663913+0000","last_peered":"2023-01-25T11:00:55.663913+0000","last_clean":"2023-01-25T11:00:55.663913+0000","last_became_active":"2023-01-25T11:00:55.662379+0000","last_became_peered":"2023-01-25T11:00:55.662379+0000","last_unstale":"2023-01-25T11:00:55.663913+0000","last_undegraded":"2023-01-25T11:00:55.663913+0000","last_fullsized":"2023-01-25T11:00:55.663913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T16:06:28.820568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.672256+0000","last_change":"2023-01-25T11:00:55.666575+0000","last_active":"2023-01-25T11:00:56.672256+0000","last_peered":"2023-01-25T11:00:56.672256+0000","last_clean":"2023-01-25T11:00:56.672256+0000","last_became_active":"2023-01-25T11:00:55.665925+0000","last_became_peered":"2023-01-25T11:00:55.665925+0000","last_unstale":"2023-01-25T11:00:56.672256+0000","last_undegraded":"2023-01-25T11:00:56.672256+0000","last_fullsized":"2023-01-25T11:00:56.672256+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T18:15:28.854851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":279584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5360,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366480,"statfs":{"total":96636764160,"available":96631275520,"internally_reserved":0,"allocated":237568,"data_stored":178725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366816,"statfs":{"total":96636764160,"available":96631619584,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5352,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366488,"statfs":{"total":96636764160,"available":96631283712,"internally_reserved":0,"allocated":229376,"data_stored":174097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4968,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366872,"statfs":{"total":96636764160,"available":96631676928,"internally_reserved":0,"allocated":98304,"data_stored":40413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T11:00:59.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T11:01:00.270 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:01:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-01-25T11:01:00.284 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-01-25T11:00:58.609832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":40544,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":754934176,"statfs":{"total":773094113280,"available":773052596224,"internally_reserved":0,"allocated":1015808,"data_stored":572160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.750869+0000","last_change":"2023-01-25T11:00:55.750203+0000","last_active":"2023-01-25T11:00:55.750869+0000","last_peered":"2023-01-25T11:00:55.750869+0000","last_clean":"2023-01-25T11:00:55.750869+0000","last_became_active":"2023-01-25T11:00:55.749892+0000","last_became_peered":"2023-01-25T11:00:55.749892+0000","last_unstale":"2023-01-25T11:00:55.750869+0000","last_undegraded":"2023-01-25T11:00:55.750869+0000","last_fullsized":"2023-01-25T11:00:55.750869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:09:11.429769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.672264+0000","last_change":"2023-01-25T11:00:55.662042+0000","last_active":"2023-01-25T11:00:56.672264+0000","last_peered":"2023-01-25T11:00:56.672264+0000","last_clean":"2023-01-25T11:00:56.672264+0000","last_became_active":"2023-01-25T11:00:55.661764+0000","last_became_peered":"2023-01-25T11:00:55.661764+0000","last_unstale":"2023-01-25T11:00:56.672264+0000","last_undegraded":"2023-01-25T11:00:56.672264+0000","last_fullsized":"2023-01-25T11:00:56.672264+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:04:49.567058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.670623+0000","last_change":"2023-01-25T11:00:55.666198+0000","last_active":"2023-01-25T11:00:56.670623+0000","last_peered":"2023-01-25T11:00:56.670623+0000","last_clean":"2023-01-25T11:00:56.670623+0000","last_became_active":"2023-01-25T11:00:55.665948+0000","last_became_peered":"2023-01-25T11:00:55.665948+0000","last_unstale":"2023-01-25T11:00:56.670623+0000","last_undegraded":"2023-01-25T11:00:56.670623+0000","last_fullsized":"2023-01-25T11:00:56.670623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:29:29.559334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.670623+0000","last_change":"2023-01-25T11:00:55.891233+0000","last_active":"2023-01-25T11:00:56.670623+0000","last_peered":"2023-01-25T11:00:56.670623+0000","last_clean":"2023-01-25T11:00:56.670623+0000","last_became_active":"2023-01-25T11:00:55.890849+0000","last_became_peered":"2023-01-25T11:00:55.890849+0000","last_unstale":"2023-01-25T11:00:56.670623+0000","last_undegraded":"2023-01-25T11:00:56.670623+0000","last_fullsized":"2023-01-25T11:00:56.670623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:22:26.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'34","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.933860+0000","last_change":"2023-01-25T11:00:54.663781+0000","last_active":"2023-01-25T11:00:56.933860+0000","last_peered":"2023-01-25T11:00:56.933860+0000","last_clean":"2023-01-25T11:00:56.933860+0000","last_became_active":"2023-01-25T11:00:54.663067+0000","last_became_peered":"2023-01-25T11:00:54.663067+0000","last_unstale":"2023-01-25T11:00:56.933860+0000","last_undegraded":"2023-01-25T11:00:56.933860+0000","last_fullsized":"2023-01-25T11:00:56.933860+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_clean_scrub_stamp":"2023-01-25T11:00:53.636416+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:09:10.422745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.660459+0000","last_change":"2023-01-25T11:00:55.659957+0000","last_active":"2023-01-25T11:00:55.660459+0000","last_peered":"2023-01-25T11:00:55.660459+0000","last_clean":"2023-01-25T11:00:55.660459+0000","last_became_active":"2023-01-25T11:00:55.659687+0000","last_became_peered":"2023-01-25T11:00:55.659687+0000","last_unstale":"2023-01-25T11:00:55.660459+0000","last_undegraded":"2023-01-25T11:00:55.660459+0000","last_fullsized":"2023-01-25T11:00:55.660459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T22:07:34.953392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.757970+0000","last_change":"2023-01-25T11:00:55.757378+0000","last_active":"2023-01-25T11:00:55.757970+0000","last_peered":"2023-01-25T11:00:55.757970+0000","last_clean":"2023-01-25T11:00:55.757970+0000","last_became_active":"2023-01-25T11:00:55.757013+0000","last_became_peered":"2023-01-25T11:00:55.757013+0000","last_unstale":"2023-01-25T11:00:55.757970+0000","last_undegraded":"2023-01-25T11:00:55.757970+0000","last_fullsized":"2023-01-25T11:00:55.757970+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:58:57.108698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-01-25T11:00:55.663913+0000","last_change":"2023-01-25T11:00:55.662604+0000","last_active":"2023-01-25T11:00:55.663913+0000","last_peered":"2023-01-25T11:00:55.663913+0000","last_clean":"2023-01-25T11:00:55.663913+0000","last_became_active":"2023-01-25T11:00:55.662379+0000","last_became_peered":"2023-01-25T11:00:55.662379+0000","last_unstale":"2023-01-25T11:00:55.663913+0000","last_undegraded":"2023-01-25T11:00:55.663913+0000","last_fullsized":"2023-01-25T11:00:55.663913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T16:06:28.820568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-01-25T11:00:56.672256+0000","last_change":"2023-01-25T11:00:55.666575+0000","last_active":"2023-01-25T11:00:56.672256+0000","last_peered":"2023-01-25T11:00:56.672256+0000","last_clean":"2023-01-25T11:00:56.672256+0000","last_became_active":"2023-01-25T11:00:55.665925+0000","last_became_peered":"2023-01-25T11:00:55.665925+0000","last_unstale":"2023-01-25T11:00:56.672256+0000","last_undegraded":"2023-01-25T11:00:56.672256+0000","last_fullsized":"2023-01-25T11:00:56.672256+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T18:15:28.854851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":60,"num_write_kb":459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":279584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5360,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366480,"statfs":{"total":96636764160,"available":96631275520,"internally_reserved":0,"allocated":237568,"data_stored":178725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5024,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366816,"statfs":{"total":96636764160,"available":96631619584,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5352,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366488,"statfs":{"total":96636764160,"available":96631283712,"internally_reserved":0,"allocated":229376,"data_stored":174097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4968,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366872,"statfs":{"total":96636764160,"available":96631676928,"internally_reserved":0,"allocated":98304,"data_stored":40413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4960,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366880,"statfs":{"total":96636764160,"available":96631685120,"internally_reserved":0,"allocated":90112,"data_stored":35785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":139792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T11:01:00.285 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-25T11:01:00.286 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-25T11:01:00.286 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-25T11:01:00.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-25T11:01:00.662 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:01:00.662 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-25T11:01:00.677 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-25T11:01:00.677 INFO:teuthology.run_tasks:Running task exec... 2023-01-25T11:01:00.687 INFO:teuthology.task.exec:Executing custom commands... 2023-01-25T11:01:00.688 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2023-01-25T11:01:00.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2023-01-25T11:01:01.021 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-01-25T11:01:01.021 INFO:teuthology.orchestra.run.smithi027.stdout: "active": true, 2023-01-25T11:01:01.022 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_duration": "0:00:00.003493", 2023-01-25T11:01:01.022 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_started": "Wed Jan 25 11:00:48 2023", 2023-01-25T11:01:01.022 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "upmap", 2023-01-25T11:01:01.022 INFO:teuthology.orchestra.run.smithi027.stdout: "no_optimization_needed": false, 2023-01-25T11:01:01.023 INFO:teuthology.orchestra.run.smithi027.stdout: "optimize_result": "No pools available", 2023-01-25T11:01:01.023 INFO:teuthology.orchestra.run.smithi027.stdout: "plans": [] 2023-01-25T11:01:01.023 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-01-25T11:01:01.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2023-01-25T11:01:01.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2023-01-25T11:01:02.138 INFO:teuthology.run_tasks:Running task full_sequential... 2023-01-25T11:01:02.147 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2023-01-25T11:01:02.149 INFO:tasks.radosbench:Beginning radosbench... 2023-01-25T11:01:02.149 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-01-25T11:01:02.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2023-01-25T11:01:03.073 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_0' created 2023-01-25T11:01:03.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-01-25T11:01:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-01-25T11:01:05.096 DEBUG:teuthology.orchestra.run.smithi027:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2023-01-25T11:01:05.101 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2023-01-25T11:01:05.102 INFO:teuthology.orchestra.run:waiting for 3000 2023-01-25T11:01:05.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:hints = 1 2023-01-25T11:01:05.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2023-01-25T11:01:05.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object prefix: benchmark_data_smithi027_111853 2023-01-25T11:01:05.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:01:05.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 0 0 0 0 0 0 - 0 2023-01-25T11:01:05.170 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:01:05.163+0000 7f4d2f1de700 -1 osd.6 20 handle_osd_map injecting map crc failure 2023-01-25T11:01:06.163 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 1 16 100 84 5.24926 5.25 0.20439 0.165315 2023-01-25T11:01:07.163 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 2 16 212 196 6.12407 7 0.109221 0.154888 2023-01-25T11:01:08.163 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 3 16 347 331 6.89478 8.4375 0.102856 0.142126 2023-01-25T11:01:09.163 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 4 16 433 417 6.51461 5.375 0.00247229 0.149018 2023-01-25T11:01:10.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 5 16 539 523 6.53657 6.625 0.102819 0.149191 2023-01-25T11:01:11.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 6 16 611 595 6.1971 4.5 0.204152 0.15612 2023-01-25T11:01:12.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 7 16 730 714 6.37417 7.4375 0.114693 0.154054 2023-01-25T11:01:13.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 8 16 826 810 6.32728 6 0.102998 0.156343 2023-01-25T11:01:14.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 9 16 944 928 6.44356 7.375 0.107235 0.153815 2023-01-25T11:01:15.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 10 16 1023 1007 6.29288 4.9375 0.204921 0.155997 2023-01-25T11:01:16.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 11 16 1103 1087 6.17531 5 0.00351045 0.158702 2023-01-25T11:01:17.164 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 12 16 1199 1183 6.16062 6 0.0031648 0.160208 2023-01-25T11:01:18.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 13 16 1294 1278 6.14339 5.9375 0.0107806 0.161708 2023-01-25T11:01:19.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 14 16 1379 1363 6.08401 5.3125 0.00251818 0.163418 2023-01-25T11:01:20.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 15 16 1456 1440 5.99921 4.8125 0.102646 0.165522 2023-01-25T11:01:21.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 16 16 1533 1517 5.92502 4.8125 0.0018615 0.166577 2023-01-25T11:01:22.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 17 16 1619 1603 5.89262 5.375 0.102433 0.168121 2023-01-25T11:01:23.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 18 16 1725 1709 5.93325 6.625 0.00225355 0.167692 2023-01-25T11:01:24.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 19 16 1809 1793 5.89725 5.25 0.00238843 0.168704 2023-01-25T11:01:25.165 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:01:25.164500+0000 min lat: 0.00152111 max lat: 0.705172 avg lat: 0.167628 2023-01-25T11:01:25.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:01:25.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 20 16 1916 1900 5.93673 6.6875 0.101956 0.167628 2023-01-25T11:01:26.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 21 16 2019 2003 5.96054 6.4375 0.102492 0.16693 2023-01-25T11:01:27.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 22 16 2100 2084 5.91969 5.0625 0.299301 0.168033 2023-01-25T11:01:28.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 23 16 2188 2172 5.90142 5.5 0.209847 0.167982 2023-01-25T11:01:29.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 24 16 2283 2267 5.90288 5.9375 0.00386383 0.168115 2023-01-25T11:01:30.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 25 16 2366 2350 5.87423 5.1875 0.100099 0.169557 2023-01-25T11:01:31.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 26 16 2463 2447 5.88145 6.0625 0.102075 0.169333 2023-01-25T11:01:32.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 27 16 2562 2546 5.89275 6.1875 0.102177 0.168998 2023-01-25T11:01:33.166 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 28 16 2636 2620 5.84746 4.625 0.396132 0.169809 2023-01-25T11:01:34.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 29 16 2723 2707 5.8333 5.4375 0.106401 0.170373 2023-01-25T11:01:35.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 30 16 2816 2800 5.83257 5.8125 0.108253 0.17075 2023-01-25T11:01:36.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 31 16 2913 2897 5.83996 6.0625 0.00595979 0.170626 2023-01-25T11:01:37.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 32 16 3016 3000 5.8586 6.4375 0.0760751 0.170126 2023-01-25T11:01:38.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 33 16 3098 3082 5.83636 5.125 0.00446261 0.171091 2023-01-25T11:01:39.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 34 16 3181 3165 5.81727 5.1875 0.18775 0.171295 2023-01-25T11:01:40.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 35 16 3265 3249 5.80104 5.25 0.20565 0.171549 2023-01-25T11:01:41.167 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 36 16 3372 3356 5.82563 6.6875 0.00794113 0.17117 2023-01-25T11:01:42.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 37 16 3461 3445 5.8185 5.5625 0.0048573 0.171568 2023-01-25T11:01:43.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 38 16 3551 3535 5.81339 5.625 0.201979 0.171392 2023-01-25T11:01:44.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 39 16 3629 3613 5.78931 4.875 0.00675738 0.171992 2023-01-25T11:01:45.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:01:45.167143+0000 min lat: 0.00152111 max lat: 0.705172 avg lat: 0.172652 2023-01-25T11:01:45.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:01:45.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 40 16 3714 3698 5.77737 5.3125 0.111468 0.172652 2023-01-25T11:01:46.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 41 16 3823 3807 5.8026 6.8125 0.0042259 0.171841 2023-01-25T11:01:47.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 42 16 3912 3896 5.79687 5.5625 0.00746633 0.172033 2023-01-25T11:01:48.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 43 16 3995 3979 5.78268 5.1875 0.102761 0.17256 2023-01-25T11:01:49.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 44 16 4084 4068 5.77767 5.5625 0.102094 0.172798 2023-01-25T11:01:50.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 45 16 4175 4159 5.77566 5.6875 0.0018668 0.172531 2023-01-25T11:01:51.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 46 16 4280 4264 5.79274 6.5625 0.323692 0.172081 2023-01-25T11:01:52.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 47 16 4378 4362 5.79979 6.125 0.00194694 0.171845 2023-01-25T11:01:53.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 48 16 4461 4445 5.78702 5.1875 0.0107222 0.172397 2023-01-25T11:01:54.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 49 16 4546 4530 5.77732 5.3125 0.00320873 0.172765 2023-01-25T11:01:55.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 50 16 4632 4616 5.76926 5.375 0.201008 0.172688 2023-01-25T11:01:56.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 51 16 4731 4715 5.77745 6.1875 0.202086 0.172367 2023-01-25T11:01:57.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 52 16 4822 4806 5.7757 5.6875 0.303448 0.172423 2023-01-25T11:01:58.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 53 16 4912 4896 5.77284 5.625 0.211752 0.172931 2023-01-25T11:01:59.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 54 16 4996 4980 5.76314 5.25 0.113782 0.173052 2023-01-25T11:02:00.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 55 16 5075 5059 5.74812 4.9375 0.111036 0.173674 2023-01-25T11:02:01.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 56 16 5154 5138 5.73364 4.9375 0.085177 0.174049 2023-01-25T11:02:02.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 57 16 5244 5228 5.73172 5.625 0.00225591 0.174011 2023-01-25T11:02:02.910 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:02:02.907+0000 7fac7dafd700 -1 osd.1 21 handle_osd_map injecting map crc failure 2023-01-25T11:02:02.911 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:02:02.909+0000 7f5f355a1700 -1 osd.0 21 handle_osd_map injecting map crc failure 2023-01-25T11:02:03.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 58 16 5329 5313 5.72448 5.3125 0.00271603 0.174435 2023-01-25T11:02:04.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 59 16 5415 5399 5.71854 5.375 0.00191359 0.174579 2023-01-25T11:02:04.862 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:02:04.859+0000 7f5f355a1700 -1 osd.0 23 handle_osd_map injecting map crc failure 2023-01-25T11:02:05.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:02:05.169599+0000 min lat: 0.00152111 max lat: 0.708959 avg lat: 0.174597 2023-01-25T11:02:05.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:02:05.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 60 16 5503 5487 5.71489 5.5 0.00273936 0.174597 2023-01-25T11:02:05.949 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:02:05.946+0000 7ffad630c700 -1 osd.4 24 handle_osd_map injecting map crc failure 2023-01-25T11:02:06.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 61 16 5588 5572 5.70828 5.3125 0.303041 0.17465 2023-01-25T11:02:07.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 62 16 5675 5659 5.70391 5.4375 0.183994 0.17487 2023-01-25T11:02:08.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 63 16 5747 5731 5.68479 4.5 0.291313 0.175387 2023-01-25T11:02:09.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 64 16 5827 5811 5.67408 5 0.142432 0.175809 2023-01-25T11:02:10.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 65 16 5937 5921 5.69254 6.875 0.00546684 0.175508 2023-01-25T11:02:11.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 66 16 6011 5995 5.67636 4.625 0.00224181 0.175775 2023-01-25T11:02:12.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 67 16 6084 6068 5.65973 4.5625 0.00247329 0.176145 2023-01-25T11:02:13.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 68 16 6178 6162 5.66288 5.875 0.00326104 0.176147 2023-01-25T11:02:14.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 69 16 6268 6252 5.66232 5.625 0.0162333 0.176304 2023-01-25T11:02:15.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 70 16 6354 6338 5.6582 5.375 0.00218564 0.176406 2023-01-25T11:02:16.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 71 16 6451 6435 5.66388 6.0625 0.00380138 0.176382 2023-01-25T11:02:17.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 72 16 6531 6515 5.65465 5 0.00506958 0.176548 2023-01-25T11:02:18.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 73 16 6633 6617 5.66451 6.375 0.00470487 0.176359 2023-01-25T11:02:19.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 74 16 6709 6693 5.65215 4.75 0.395454 0.17656 2023-01-25T11:02:20.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 75 16 6788 6772 5.64261 4.9375 0.00471301 0.17687 2023-01-25T11:02:21.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 76 16 6885 6869 5.64813 6.0625 0.00348951 0.176832 2023-01-25T11:02:22.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 77 16 6972 6956 5.64538 5.4375 0.0915833 0.176907 2023-01-25T11:02:23.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 78 16 7061 7045 5.64431 5.5625 0.0115768 0.176944 2023-01-25T11:02:24.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 79 16 7137 7121 5.63298 4.75 0.302738 0.177352 2023-01-25T11:02:25.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:02:25.172171+0000 min lat: 0.00152111 max lat: 0.708959 avg lat: 0.177232 2023-01-25T11:02:25.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:02:25.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 80 16 7226 7210 5.63209 5.5625 0.00175789 0.177232 2023-01-25T11:02:26.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 81 16 7326 7310 5.6397 6.25 0.0236928 0.177084 2023-01-25T11:02:27.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 82 16 7411 7395 5.63571 5.3125 0.00191563 0.177039 2023-01-25T11:02:28.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 83 16 7504 7488 5.63782 5.8125 0.00502649 0.177243 2023-01-25T11:02:29.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 84 16 7601 7585 5.64287 6.0625 0.00220928 0.176855 2023-01-25T11:02:30.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 85 16 7680 7664 5.63457 4.9375 0.00301512 0.177176 2023-01-25T11:02:31.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 86 16 7760 7744 5.62718 5 0.104693 0.17727 2023-01-25T11:02:32.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 87 16 7818 7802 5.60416 3.625 0.0045626 0.177883 2023-01-25T11:02:33.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 88 16 7899 7883 5.598 5.0625 0.202608 0.178274 2023-01-25T11:02:34.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 89 16 7978 7962 5.59057 4.9375 0.002298 0.178707 2023-01-25T11:02:35.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 90 16 8062 8046 5.58678 5.25 0.103361 0.178698 2023-01-25T11:02:36.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total time run: 90.2155 2023-01-25T11:02:36.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total writes made: 8062 2023-01-25T11:02:36.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Write size: 65536 2023-01-25T11:02:36.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object size: 65536 2023-01-25T11:02:36.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Bandwidth (MB/sec): 5.58524 2023-01-25T11:02:36.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Bandwidth: 0.747585 2023-01-25T11:02:36.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max bandwidth (MB/sec): 8.4375 2023-01-25T11:02:36.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min bandwidth (MB/sec): 3.625 2023-01-25T11:02:36.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average IOPS: 89 2023-01-25T11:02:36.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev IOPS: 11.9614 2023-01-25T11:02:36.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max IOPS: 135 2023-01-25T11:02:36.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min IOPS: 58 2023-01-25T11:02:36.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average Latency(s): 0.178791 2023-01-25T11:02:36.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Latency(s): 0.129031 2023-01-25T11:02:36.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max latency(s): 0.792821 2023-01-25T11:02:36.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min latency(s): 0.00152111 2023-01-25T11:02:36.578 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Cleaning up (deleting benchmark objects) 2023-01-25T11:03:03.792 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:03:03.791+0000 7fac7fb01700 -1 osd.1 26 handle_osd_map injecting map crc failure 2023-01-25T11:03:04.800 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:03:04.797+0000 7fb8a093d700 -1 osd.7 27 handle_osd_map injecting map crc failure 2023-01-25T11:03:04.918 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:03:04.916+0000 7f4d2d1da700 -1 osd.6 27 handle_osd_map injecting map crc failure 2023-01-25T11:04:03.921 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:04:03.918+0000 7f5f3359d700 -1 osd.0 31 handle_osd_map injecting map crc failure 2023-01-25T11:04:04.948 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:04:04.946+0000 7fa985af6700 -1 osd.5 32 handle_osd_map injecting map crc failure 2023-01-25T11:04:05.043 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:04:05.040+0000 7f4d2d1da700 -1 osd.6 32 handle_osd_map injecting map crc failure 2023-01-25T11:04:05.047 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:04:05.045+0000 7fac7dafd700 -1 osd.1 32 handle_osd_map injecting map crc failure 2023-01-25T11:04:07.888 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Removed 8062 objects 2023-01-25T11:04:07.888 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Clean up completed and total clean up time :91.3094 2023-01-25T11:04:08.145 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2023-01-25T11:04:08.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2023-01-25T11:04:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_0' removed 2023-01-25T11:04:08.938 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2023-01-25T11:04:08.939 INFO:tasks.radosbench:Beginning radosbench... 2023-01-25T11:04:08.940 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_1 2023-01-25T11:04:08.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_1 16 2023-01-25T11:04:09.923 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:04:09.920+0000 7fac7fb01700 -1 osd.1 34 handle_osd_map injecting map crc failure 2023-01-25T11:04:09.924 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:04:09.922+0000 7fa987afa700 -1 osd.5 34 handle_osd_map injecting map crc failure 2023-01-25T11:04:09.929 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_1' created 2023-01-25T11:04:09.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2023-01-25T11:04:09.974 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:04:09.972+0000 7ffad8310700 -1 osd.4 33 handle_osd_map injecting map crc failure 2023-01-25T11:04:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_1' 2023-01-25T11:04:11.931 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:04:11.929+0000 7fc3c94ae700 -1 osd.2 36 handle_osd_map injecting map crc failure 2023-01-25T11:04:11.953 DEBUG:teuthology.orchestra.run.smithi027:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2023-01-25T11:04:11.956 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2023-01-25T11:04:11.956 INFO:teuthology.orchestra.run:waiting for 3000 2023-01-25T11:04:12.014 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:hints = 1 2023-01-25T11:04:12.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2023-01-25T11:04:12.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object prefix: benchmark_data_smithi027_112081 2023-01-25T11:04:12.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:04:12.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 0 0 0 0 0 0 - 0 2023-01-25T11:04:12.016 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:04:12.013+0000 7fac7fb01700 -1 osd.1 36 handle_osd_map injecting map crc failure 2023-01-25T11:04:13.014 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 1 16 98 82 5.1247 5.125 0.111547 0.163674 2023-01-25T11:04:14.014 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 2 16 200 184 5.7496 6.375 0.00504181 0.161692 2023-01-25T11:04:15.014 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 3 16 304 288 5.99949 6.5 0.00614534 0.160891 2023-01-25T11:04:16.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 4 16 391 375 5.85886 5.4375 0.00266665 0.16694 2023-01-25T11:04:17.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 5 16 484 468 5.84947 5.8125 0.00910185 0.167996 2023-01-25T11:04:18.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 6 16 570 554 5.77026 5.375 0.00185286 0.16792 2023-01-25T11:04:19.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 7 16 661 645 5.75836 5.6875 0.103985 0.170199 2023-01-25T11:04:20.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 8 16 765 749 5.85098 6.5 0.106496 0.168693 2023-01-25T11:04:21.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 9 16 857 841 5.83969 5.75 0.102337 0.168761 2023-01-25T11:04:22.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 10 16 951 935 5.84318 5.875 0.101976 0.168328 2023-01-25T11:04:23.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 11 15 1033 1018 5.78353 5.1875 0.40324 0.170002 2023-01-25T11:04:24.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 12 16 1123 1107 5.76508 5.5625 0.103181 0.171107 2023-01-25T11:04:25.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 13 16 1205 1189 5.71581 5.125 0.10496 0.172628 2023-01-25T11:04:26.015 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 14 16 1301 1285 5.73608 6 0.102433 0.172679 2023-01-25T11:04:27.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 15 16 1401 1385 5.77031 6.25 0.205377 0.17159 2023-01-25T11:04:28.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 16 16 1496 1480 5.78072 5.9375 0.00215563 0.171826 2023-01-25T11:04:29.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 17 16 1584 1568 5.76418 5.5 0.202174 0.172258 2023-01-25T11:04:30.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 18 16 1674 1658 5.75642 5.625 0.00284809 0.171636 2023-01-25T11:04:31.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 19 16 1770 1754 5.76921 6 0.013086 0.171938 2023-01-25T11:04:32.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:04:32.015277+0000 min lat: 0.0014439 max lat: 0.708869 avg lat: 0.170657 2023-01-25T11:04:32.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:04:32.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 20 16 1885 1869 5.84009 7.1875 0.00430176 0.170657 2023-01-25T11:04:33.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 21 16 1980 1964 5.84469 5.9375 0.0019463 0.169979 2023-01-25T11:04:34.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 22 16 2080 2064 5.86309 6.25 0.100839 0.169754 2023-01-25T11:04:35.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 23 16 2177 2161 5.87174 6.0625 0.208021 0.169911 2023-01-25T11:04:36.016 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 24 16 2258 2242 5.83799 5.0625 0.102649 0.170553 2023-01-25T11:04:37.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 25 16 2363 2347 5.86694 6.5625 0.20233 0.169598 2023-01-25T11:04:38.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 26 16 2468 2452 5.89366 6.5625 0.208647 0.16881 2023-01-25T11:04:39.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 27 16 2557 2541 5.88138 5.5625 0.104762 0.169523 2023-01-25T11:04:40.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 28 16 2651 2635 5.88113 5.875 0.206707 0.169734 2023-01-25T11:04:41.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 29 16 2742 2726 5.87444 5.6875 0.00252036 0.169518 2023-01-25T11:04:42.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 30 16 2847 2831 5.89735 6.5625 0.00425882 0.168998 2023-01-25T11:04:43.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 31 16 2928 2912 5.87041 5.0625 0.102573 0.169886 2023-01-25T11:04:44.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 32 16 3013 2997 5.85296 5.3125 0.00285878 0.169926 2023-01-25T11:04:45.017 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 33 16 3103 3087 5.84603 5.625 0.101761 0.170101 2023-01-25T11:04:46.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 34 16 3195 3179 5.84318 5.75 0.102894 0.170276 2023-01-25T11:04:47.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 35 16 3287 3271 5.8405 5.75 0.302454 0.170152 2023-01-25T11:04:48.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 36 16 3382 3366 5.84317 5.9375 0.105949 0.170626 2023-01-25T11:04:49.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 37 16 3484 3468 5.85752 6.375 0.102245 0.170008 2023-01-25T11:04:50.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 38 16 3574 3558 5.85138 5.625 0.00183121 0.170251 2023-01-25T11:04:51.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 39 16 3654 3638 5.82954 5 0.199735 0.170941 2023-01-25T11:04:52.018 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:04:52.017441+0000 min lat: 0.0014439 max lat: 0.708869 avg lat: 0.170919 2023-01-25T11:04:52.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:04:52.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 40 16 3748 3732 5.83067 5.875 0.103242 0.170919 2023-01-25T11:04:53.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 41 16 3842 3826 5.83173 5.875 0.0019447 0.170755 2023-01-25T11:04:54.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 42 16 3930 3914 5.82381 5.5 0.109227 0.171142 2023-01-25T11:04:55.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 43 16 4040 4024 5.84824 6.875 0.114035 0.170622 2023-01-25T11:04:56.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 44 16 4138 4122 5.85451 6.125 0.204862 0.170566 2023-01-25T11:04:57.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 45 16 4218 4202 5.83551 5 0.100858 0.17059 2023-01-25T11:04:58.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 46 16 4312 4296 5.83635 5.875 0.0118278 0.171069 2023-01-25T11:04:59.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 47 16 4410 4394 5.84247 6.125 0.00205037 0.170628 2023-01-25T11:05:00.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 48 16 4503 4487 5.84183 5.8125 0.120896 0.170532 2023-01-25T11:05:01.019 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 49 16 4594 4578 5.83867 5.6875 0.0986525 0.171033 2023-01-25T11:05:02.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 50 16 4703 4687 5.85813 6.8125 0.00431339 0.170239 2023-01-25T11:05:03.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 51 16 4795 4779 5.85599 5.75 0.00317887 0.170132 2023-01-25T11:05:04.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 52 16 4895 4879 5.86356 6.25 0.10115 0.170044 2023-01-25T11:05:04.182 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:05:04.181+0000 7f5f3359d700 -1 osd.0 37 handle_osd_map injecting map crc failure 2023-01-25T11:05:05.020 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:05:05.018+0000 7fac7fb01700 -1 osd.1 38 handle_osd_map injecting map crc failure 2023-01-25T11:05:05.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 53 16 4989 4973 5.86376 5.875 0.00238612 0.169914 2023-01-25T11:05:05.022 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:05:05.021+0000 7f5f355a1700 -1 osd.0 38 handle_osd_map injecting map crc failure 2023-01-25T11:05:06.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 54 16 5057 5041 5.83387 4.25 0.0882855 0.169391 2023-01-25T11:05:06.029 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:05:06.026+0000 7ffad8310700 -1 osd.4 39 handle_osd_map injecting map crc failure 2023-01-25T11:05:07.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 55 16 5148 5132 5.8312 5.6875 0.0044638 0.1712 2023-01-25T11:05:08.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 56 16 5230 5214 5.81857 5.125 0.00372037 0.171493 2023-01-25T11:05:09.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 57 16 5326 5310 5.82174 6 0.202258 0.171516 2023-01-25T11:05:10.020 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 58 16 5410 5394 5.81188 5.25 0.0940406 0.171808 2023-01-25T11:05:11.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 59 16 5490 5474 5.79811 5 0.103305 0.171719 2023-01-25T11:05:12.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:05:12.019873+0000 min lat: 0.0014439 max lat: 2.27069 avg lat: 0.172221 2023-01-25T11:05:12.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:05:12.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 60 16 5577 5561 5.79209 5.4375 0.102453 0.172221 2023-01-25T11:05:13.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 61 16 5663 5647 5.78523 5.375 0.51392 0.172125 2023-01-25T11:05:14.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 62 16 5766 5750 5.79574 6.4375 0.00291212 0.172177 2023-01-25T11:05:15.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 63 16 5853 5837 5.79005 5.4375 0.00229947 0.172467 2023-01-25T11:05:16.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 64 16 5930 5914 5.77476 4.8125 0.202221 0.172906 2023-01-25T11:05:17.021 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 65 16 6017 6001 5.76957 5.4375 0.00236476 0.172936 2023-01-25T11:05:18.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 66 16 6096 6080 5.75695 4.9375 0.103602 0.173328 2023-01-25T11:05:19.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 67 16 6212 6196 5.77922 7.25 0.11476 0.172654 2023-01-25T11:05:20.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 68 16 6310 6294 5.78429 6.125 0.00267768 0.172539 2023-01-25T11:05:20.054 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:05:20.052+0000 7fb8a2941700 -1 osd.7 40 handle_osd_map injecting map crc failure 2023-01-25T11:05:20.251 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:05:20.249+0000 7fa985af6700 -1 osd.5 40 handle_osd_map injecting map crc failure 2023-01-25T11:05:21.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 69 16 6387 6371 5.7702 4.8125 0.209923 0.172771 2023-01-25T11:05:22.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 70 16 6471 6455 5.76275 5.25 0.103501 0.173165 2023-01-25T11:05:23.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 71 16 6583 6567 5.78017 7 0.00288805 0.172828 2023-01-25T11:05:24.022 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 72 16 6681 6665 5.78495 6.125 0.00239548 0.172715 2023-01-25T11:05:25.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 73 16 6766 6750 5.77847 5.3125 0.00204251 0.172622 2023-01-25T11:05:26.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 74 16 6819 6803 5.74513 3.3125 0.108851 0.173532 2023-01-25T11:05:27.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 75 16 6897 6881 5.73352 4.875 0.00393169 0.174064 2023-01-25T11:05:28.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 76 16 6993 6977 5.73702 6 0.00237426 0.174128 2023-01-25T11:05:29.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 77 16 7082 7066 5.73474 5.5625 0.00194084 0.174117 2023-01-25T11:05:30.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 78 16 7181 7165 5.74054 6.1875 0.10877 0.174029 2023-01-25T11:05:31.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 79 16 7270 7254 5.73828 5.5625 0.00635787 0.173961 2023-01-25T11:05:32.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:05:32.022466+0000 min lat: 0.0014439 max lat: 2.27069 avg lat: 0.174186 2023-01-25T11:05:32.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:05:32.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 80 16 7354 7338 5.73217 5.25 0.175797 0.174186 2023-01-25T11:05:33.023 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 81 16 7445 7429 5.73161 5.6875 0.00386575 0.174296 2023-01-25T11:05:34.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 82 16 7542 7526 5.73563 6.0625 0.00584903 0.174101 2023-01-25T11:05:35.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 83 16 7644 7628 5.74333 6.375 0.00206325 0.173957 2023-01-25T11:05:36.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 84 16 7734 7718 5.74191 5.625 0.0974795 0.174007 2023-01-25T11:05:37.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 85 16 7818 7802 5.73612 5.25 0.0949753 0.174074 2023-01-25T11:05:38.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 86 16 7911 7895 5.73699 5.8125 0.00556796 0.174029 2023-01-25T11:05:39.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 87 16 8002 7986 5.73642 5.6875 0.102789 0.17397 2023-01-25T11:05:40.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 88 16 8079 8063 5.72591 4.8125 0.102012 0.174395 2023-01-25T11:05:41.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 89 16 8183 8167 5.73461 6.5 0.00911732 0.174244 2023-01-25T11:05:42.024 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 90 16 8266 8250 5.72852 5.1875 0.00246103 0.174426 2023-01-25T11:05:43.025 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total time run: 90.2968 2023-01-25T11:05:43.025 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total writes made: 8266 2023-01-25T11:05:43.025 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Write size: 65536 2023-01-25T11:05:43.025 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object size: 65536 2023-01-25T11:05:43.026 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Bandwidth (MB/sec): 5.72141 2023-01-25T11:05:43.026 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Bandwidth: 0.625351 2023-01-25T11:05:43.026 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max bandwidth (MB/sec): 7.25 2023-01-25T11:05:43.026 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min bandwidth (MB/sec): 3.3125 2023-01-25T11:05:43.026 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average IOPS: 91 2023-01-25T11:05:43.027 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev IOPS: 10.0056 2023-01-25T11:05:43.027 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max IOPS: 116 2023-01-25T11:05:43.027 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min IOPS: 53 2023-01-25T11:05:43.027 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average Latency(s): 0.174554 2023-01-25T11:05:43.027 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Latency(s): 0.131517 2023-01-25T11:05:43.028 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max latency(s): 2.27069 2023-01-25T11:05:43.028 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min latency(s): 0.0014439 2023-01-25T11:05:43.129 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Cleaning up (deleting benchmark objects) 2023-01-25T11:06:04.875 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:06:04.871+0000 7fa985af6700 -1 osd.5 42 handle_osd_map injecting map crc failure 2023-01-25T11:06:05.114 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:06:05.112+0000 7f4d2d1da700 -1 osd.6 42 handle_osd_map injecting map crc failure 2023-01-25T11:06:05.868 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:06:05.866+0000 7f4d2f1de700 -1 osd.6 43 handle_osd_map injecting map crc failure 2023-01-25T11:06:06.883 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:06:06.880+0000 7fac7fb01700 -1 osd.1 44 handle_osd_map injecting map crc failure 2023-01-25T11:06:06.937 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:06:06.934+0000 7fb8a093d700 -1 osd.7 44 handle_osd_map injecting map crc failure 2023-01-25T11:06:25.113 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:06:25.111+0000 7fb8a093d700 -1 osd.7 45 handle_osd_map injecting map crc failure 2023-01-25T11:06:25.923 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:06:25.921+0000 7fa985af6700 -1 osd.5 46 handle_osd_map injecting map crc failure 2023-01-25T11:06:25.942 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:06:25.939+0000 7f5f3359d700 -1 osd.0 46 handle_osd_map injecting map crc failure 2023-01-25T11:07:04.985 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:07:04.983+0000 7fac7fb01700 -1 osd.1 47 handle_osd_map injecting map crc failure 2023-01-25T11:07:17.229 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Removed 8266 objects 2023-01-25T11:07:17.230 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Clean up completed and total clean up time :94.1006 2023-01-25T11:07:21.106 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_1 2023-01-25T11:07:21.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2023-01-25T11:07:22.038 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_1' removed 2023-01-25T11:07:22.051 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2023-01-25T11:07:22.052 INFO:tasks.radosbench:Beginning radosbench... 2023-01-25T11:07:22.053 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_2 2023-01-25T11:07:22.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_2 16 2023-01-25T11:07:22.074 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:07:22.072+0000 7f5f3359d700 -1 osd.0 49 handle_osd_map injecting map crc failure 2023-01-25T11:07:23.041 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_2' created 2023-01-25T11:07:23.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2023-01-25T11:07:24.045 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:07:24.042+0000 7fb8a2941700 -1 osd.7 51 handle_osd_map injecting map crc failure 2023-01-25T11:07:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_2' 2023-01-25T11:07:25.047 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:07:25.045+0000 7fac7fb01700 -1 osd.1 52 handle_osd_map injecting map crc failure 2023-01-25T11:07:25.066 DEBUG:teuthology.orchestra.run.smithi027:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2023-01-25T11:07:25.069 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2023-01-25T11:07:25.069 INFO:teuthology.orchestra.run:waiting for 3000 2023-01-25T11:07:25.130 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:hints = 1 2023-01-25T11:07:25.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2023-01-25T11:07:25.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object prefix: benchmark_data_smithi027_112303 2023-01-25T11:07:25.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:07:25.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 0 0 0 0 0 0 - 0 2023-01-25T11:07:26.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 1 16 112 96 5.99927 6 0.00539052 0.136445 2023-01-25T11:07:27.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 2 16 204 188 5.87437 5.75 0.00970287 0.150391 2023-01-25T11:07:28.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 3 16 291 275 5.72852 5.4375 0.109673 0.170505 2023-01-25T11:07:29.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 4 16 372 356 5.56187 5.0625 0.00259754 0.168776 2023-01-25T11:07:30.131 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 5 16 467 451 5.63688 5.9375 0.00526653 0.174484 2023-01-25T11:07:31.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 6 16 572 556 5.79106 6.5625 0.00603288 0.170721 2023-01-25T11:07:32.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 7 16 643 627 5.59762 4.4375 0.197021 0.172963 2023-01-25T11:07:33.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 8 16 750 734 5.73376 6.6875 0.210119 0.171981 2023-01-25T11:07:34.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 9 16 876 860 5.97157 7.875 0.00184239 0.165934 2023-01-25T11:07:35.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 10 16 991 975 6.0931 7.1875 0.108624 0.162559 2023-01-25T11:07:36.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 11 16 1083 1067 6.06186 5.75 0.00219235 0.161687 2023-01-25T11:07:37.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 12 16 1183 1167 6.07749 6.25 0.00193381 0.162225 2023-01-25T11:07:38.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 13 16 1271 1255 6.03302 5.5 0.704323 0.163131 2023-01-25T11:07:39.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 14 16 1364 1348 6.01721 5.8125 0.105564 0.164648 2023-01-25T11:07:40.132 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 15 16 1461 1445 6.02019 6.0625 0.103497 0.165011 2023-01-25T11:07:41.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 16 16 1560 1544 6.03061 6.1875 0.305252 0.164846 2023-01-25T11:07:42.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 17 16 1634 1618 5.9479 4.625 0.106891 0.166469 2023-01-25T11:07:43.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 18 16 1733 1717 5.96116 6.1875 0.00201894 0.166592 2023-01-25T11:07:44.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 19 16 1821 1805 5.93684 5.5 0.00594216 0.167549 2023-01-25T11:07:45.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:07:45.132370+0000 min lat: 0.00161038 max lat: 0.712 avg lat: 0.166664 2023-01-25T11:07:45.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:07:45.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 20 16 1921 1905 5.95246 6.25 0.00203927 0.166664 2023-01-25T11:07:46.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 21 16 2036 2020 6.01123 7.1875 0.00241274 0.165848 2023-01-25T11:07:47.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 22 16 2140 2124 6.03342 6.5 0.00978963 0.165106 2023-01-25T11:07:48.133 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 23 16 2231 2215 6.01836 5.6875 0.306887 0.165675 2023-01-25T11:07:49.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 24 16 2306 2290 5.96289 4.6875 0.114361 0.166784 2023-01-25T11:07:50.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 25 16 2393 2377 5.94184 5.4375 0.00430053 0.166871 2023-01-25T11:07:51.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 26 16 2504 2488 5.9801 6.9375 0.00179593 0.166582 2023-01-25T11:07:52.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 27 16 2622 2606 6.03174 7.375 0.00181941 0.165091 2023-01-25T11:07:53.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 28 16 2711 2695 6.01494 5.5625 0.00301885 0.165689 2023-01-25T11:07:54.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 29 16 2828 2812 6.05965 7.3125 0.00337344 0.164642 2023-01-25T11:07:55.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 30 16 2902 2886 6.01182 4.625 0.206598 0.166113 2023-01-25T11:07:56.134 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 31 16 2989 2973 5.99327 5.4375 0.00216547 0.165944 2023-01-25T11:07:57.135 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 32 16 3072 3056 5.96806 5.1875 0.00249079 0.166831 2023-01-25T11:07:58.135 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 33 16 3157 3141 5.94817 5.3125 0.108156 0.167336 2023-01-25T11:07:59.135 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 34 16 3240 3224 5.92577 5.1875 0.108398 0.168105 2023-01-25T11:08:00.135 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 35 16 3340 3324 5.93501 6.25 0.204524 0.16793 2023-01-25T11:08:01.135 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 36 16 3438 3422 5.94027 6.125 0.00173304 0.168062 2023-01-25T11:08:02.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 37 16 3539 3523 5.95032 6.3125 0.094852 0.167619 2023-01-25T11:08:03.135 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 38 16 3625 3609 5.93516 5.375 0.102271 0.167545 2023-01-25T11:08:04.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 39 16 3712 3696 5.92238 5.4375 0.00456879 0.168134 2023-01-25T11:08:05.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:08:05.134817+0000 min lat: 0.00161038 max lat: 0.801911 avg lat: 0.167956 2023-01-25T11:08:05.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:08:05.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 40 16 3819 3803 5.94149 6.6875 0.102523 0.167956 2023-01-25T11:08:06.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 41 16 3914 3898 5.94138 5.9375 0.104398 0.168001 2023-01-25T11:08:07.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 42 16 4022 4006 5.96061 6.75 0.202958 0.167062 2023-01-25T11:08:08.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 43 16 4143 4127 5.99784 7.5625 0.00272479 0.166242 2023-01-25T11:08:09.138 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 44 16 4258 4242 6.02485 7.1875 0.00478752 0.165629 2023-01-25T11:08:10.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 45 16 4357 4341 6.02846 6.1875 0.0982324 0.16548 2023-01-25T11:08:11.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 46 15 4452 4437 6.02782 6 0.293125 0.165371 2023-01-25T11:08:12.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 47 16 4548 4532 6.02589 5.9375 0.0048639 0.165296 2023-01-25T11:08:13.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 48 16 4661 4645 6.04747 7.0625 0.00192752 0.1649 2023-01-25T11:08:14.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 49 16 4745 4729 6.03118 5.25 0.00391932 0.165525 2023-01-25T11:08:15.136 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 50 16 4843 4827 6.03305 6.125 0.0023649 0.165577 2023-01-25T11:08:16.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 51 16 4925 4909 6.01523 5.125 0.108182 0.165775 2023-01-25T11:08:17.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 52 16 5030 5014 6.02575 6.5625 0.217975 0.16552 2023-01-25T11:08:18.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 53 16 5123 5107 6.02172 5.8125 0.20158 0.1655 2023-01-25T11:08:19.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 54 16 5223 5207 6.02593 6.25 0.00212859 0.16562 2023-01-25T11:08:20.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 55 16 5322 5306 6.02885 6.1875 0.00295735 0.165559 2023-01-25T11:08:21.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 56 16 5410 5394 6.0194 5.5 0.102535 0.165905 2023-01-25T11:08:22.137 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 57 16 5475 5459 5.98507 4.0625 0.204776 0.166597 2023-01-25T11:08:23.138 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 58 16 5558 5542 5.97126 5.1875 0.00241901 0.167023 2023-01-25T11:08:24.138 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 59 16 5645 5629 5.9622 5.4375 0.0023405 0.167346 2023-01-25T11:08:25.138 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:08:25.137377+0000 min lat: 0.0015655 max lat: 0.803427 avg lat: 0.167259 2023-01-25T11:08:25.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:08:25.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 60 16 5742 5726 5.96386 6.0625 0.00224027 0.167259 2023-01-25T11:08:26.138 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 61 16 5842 5826 5.96854 6.25 0.00210966 0.167059 2023-01-25T11:08:27.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 62 16 5940 5924 5.97106 6.125 0.00255491 0.167178 2023-01-25T11:08:28.138 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 63 16 6034 6018 5.96953 5.875 0.00431525 0.167335 2023-01-25T11:08:29.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 64 16 6105 6089 5.94558 4.4375 0.202861 0.16779 2023-01-25T11:08:30.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 65 16 6219 6203 5.96371 7.125 0.115873 0.167156 2023-01-25T11:08:31.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 66 16 6340 6324 5.98792 7.5625 0.121989 0.166589 2023-01-25T11:08:32.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 67 16 6434 6418 5.98622 5.875 0.206498 0.16671 2023-01-25T11:08:33.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 68 16 6534 6518 5.99009 6.25 0.104543 0.166793 2023-01-25T11:08:34.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 69 16 6628 6612 5.98842 5.875 0.102298 0.166646 2023-01-25T11:08:35.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 70 16 6737 6721 6.00018 6.8125 0.109636 0.166414 2023-01-25T11:08:36.139 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 71 16 6831 6815 5.9984 5.875 0.203534 0.166295 2023-01-25T11:08:37.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 72 16 6926 6910 5.99755 5.9375 0.198123 0.166302 2023-01-25T11:08:38.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 73 16 7030 7014 6.00442 6.5 0.00200559 0.166466 2023-01-25T11:08:39.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 74 16 7124 7108 6.00266 5.875 0.00214292 0.166297 2023-01-25T11:08:40.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 75 16 7211 7195 5.99512 5.4375 0.047928 0.166404 2023-01-25T11:08:41.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 76 16 7313 7297 6.0001 6.375 0.413013 0.166363 2023-01-25T11:08:42.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 77 16 7392 7376 5.9863 4.9375 0.102385 0.166793 2023-01-25T11:08:43.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 78 16 7502 7486 5.99768 6.875 0.202655 0.166464 2023-01-25T11:08:44.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 79 16 7586 7570 5.98821 5.25 0.00387174 0.166743 2023-01-25T11:08:45.140 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:08:45.139633+0000 min lat: 0.0015655 max lat: 0.803427 avg lat: 0.166966 2023-01-25T11:08:45.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:08:45.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 80 16 7671 7655 5.97976 5.3125 0.103122 0.166966 2023-01-25T11:08:46.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 81 16 7764 7748 5.97769 5.8125 0.00348513 0.167046 2023-01-25T11:08:47.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 82 16 7865 7849 5.98176 6.3125 0.0062128 0.16687 2023-01-25T11:08:48.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 83 16 7957 7941 5.97896 5.75 0.211395 0.167058 2023-01-25T11:08:49.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 84 16 8071 8055 5.99259 7.125 0.207898 0.166703 2023-01-25T11:08:50.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 85 16 8162 8146 5.98899 5.6875 0.105818 0.166666 2023-01-25T11:08:51.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 86 16 8249 8233 5.98257 5.4375 0.00491277 0.166873 2023-01-25T11:08:52.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 87 16 8340 8324 5.97916 5.6875 0.00496111 0.166975 2023-01-25T11:08:53.141 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 88 16 8427 8411 5.973 5.4375 0.104727 0.16701 2023-01-25T11:08:54.142 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 89 16 8517 8501 5.96908 5.625 0.0017167 0.167268 2023-01-25T11:08:55.142 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 90 14 8612 8598 5.97011 6.0625 0.102601 0.167228 2023-01-25T11:08:56.142 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total time run: 90.3062 2023-01-25T11:08:56.142 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total writes made: 8612 2023-01-25T11:08:56.142 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Write size: 65536 2023-01-25T11:08:56.143 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object size: 65536 2023-01-25T11:08:56.143 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Bandwidth (MB/sec): 5.96027 2023-01-25T11:08:56.143 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Bandwidth: 0.759433 2023-01-25T11:08:56.143 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max bandwidth (MB/sec): 7.875 2023-01-25T11:08:56.144 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min bandwidth (MB/sec): 4.0625 2023-01-25T11:08:56.144 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average IOPS: 95 2023-01-25T11:08:56.144 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev IOPS: 12.1509 2023-01-25T11:08:56.144 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max IOPS: 126 2023-01-25T11:08:56.144 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min IOPS: 65 2023-01-25T11:08:56.145 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average Latency(s): 0.167345 2023-01-25T11:08:56.145 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Latency(s): 0.124961 2023-01-25T11:08:56.145 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max latency(s): 0.803427 2023-01-25T11:08:56.145 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min latency(s): 0.0015655 2023-01-25T11:08:56.248 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Cleaning up (deleting benchmark objects) 2023-01-25T11:10:29.543 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Removed 8612 objects 2023-01-25T11:10:29.543 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Clean up completed and total clean up time :93.2943 2023-01-25T11:10:34.215 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_2 2023-01-25T11:10:34.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2023-01-25T11:10:35.071 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:10:35.069+0000 7fcc32ee8700 -1 osd.3 53 handle_osd_map injecting map crc failure 2023-01-25T11:10:35.078 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_2' removed 2023-01-25T11:10:35.092 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2023-01-25T11:10:35.093 INFO:tasks.radosbench:Beginning radosbench... 2023-01-25T11:10:35.093 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_3 2023-01-25T11:10:35.093 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_3 16 2023-01-25T11:10:35.253 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:10:35.251+0000 7fc3c74aa700 -1 osd.2 53 handle_osd_map injecting map crc failure 2023-01-25T11:10:35.254 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:10:35.250+0000 7ffad630c700 -1 osd.4 53 handle_osd_map injecting map crc failure 2023-01-25T11:10:36.072 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:10:36.069+0000 7fcc32ee8700 -1 osd.3 54 handle_osd_map injecting map crc failure 2023-01-25T11:10:36.072 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-01-25T11:10:36.069+0000 7fc3c94ae700 -1 osd.2 54 handle_osd_map injecting map crc failure 2023-01-25T11:10:36.073 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:10:36.069+0000 7fb8a2941700 -1 osd.7 54 handle_osd_map injecting map crc failure 2023-01-25T11:10:36.082 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_3' created 2023-01-25T11:10:36.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2023-01-25T11:10:37.082 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:10:37.078+0000 7fcc32ee8700 -1 osd.3 55 handle_osd_map injecting map crc failure 2023-01-25T11:10:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_3' 2023-01-25T11:10:37.090 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:10:37.088+0000 7ffad8310700 -1 osd.4 55 handle_osd_map injecting map crc failure 2023-01-25T11:10:38.105 DEBUG:teuthology.orchestra.run.smithi027:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2023-01-25T11:10:38.107 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2023-01-25T11:10:38.108 INFO:teuthology.orchestra.run:waiting for 3000 2023-01-25T11:10:38.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:hints = 1 2023-01-25T11:10:38.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2023-01-25T11:10:38.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object prefix: benchmark_data_smithi027_112531 2023-01-25T11:10:38.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:10:38.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 0 0 0 0 0 0 - 0 2023-01-25T11:10:39.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 1 16 104 88 5.49965 5.5 0.00223107 0.146048 2023-01-25T11:10:40.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 2 16 180 164 5.12445 4.75 0.103861 0.180257 2023-01-25T11:10:41.168 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 3 16 264 248 5.16606 5.25 0.205668 0.181631 2023-01-25T11:10:42.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 4 16 365 349 5.45245 6.3125 0.00187298 0.176943 2023-01-25T11:10:43.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 5 16 479 463 5.78678 7.125 0.10224 0.169846 2023-01-25T11:10:44.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 6 16 566 550 5.72845 5.4375 0.00374705 0.171846 2023-01-25T11:10:45.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 7 16 670 654 5.8386 6.5 0.100728 0.169033 2023-01-25T11:10:46.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 8 16 749 733 5.72593 4.9375 0.304482 0.168702 2023-01-25T11:10:47.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 9 16 857 841 5.8396 6.75 0.106699 0.169443 2023-01-25T11:10:48.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 10 16 956 940 5.8743 6.1875 0.102081 0.168315 2023-01-25T11:10:49.169 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 11 16 1065 1049 5.95954 6.8125 0.202921 0.166489 2023-01-25T11:10:50.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 12 16 1169 1153 6.00449 6.5 0.00543376 0.164705 2023-01-25T11:10:51.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 13 16 1269 1253 6.02331 6.25 0.00379731 0.16307 2023-01-25T11:10:52.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 14 16 1360 1344 5.99926 5.6875 0.00190875 0.165626 2023-01-25T11:10:53.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 15 16 1431 1415 5.8951 4.4375 0.00494588 0.16874 2023-01-25T11:10:54.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 16 16 1516 1500 5.85864 5.3125 0.200877 0.169241 2023-01-25T11:10:55.170 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 17 16 1596 1580 5.8081 5 0.102184 0.170762 2023-01-25T11:10:56.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 18 16 1682 1666 5.78399 5.375 0.00453473 0.171237 2023-01-25T11:10:57.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 19 16 1776 1760 5.78873 5.875 0.0988657 0.171119 2023-01-25T11:10:58.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:10:58.169999+0000 min lat: 0.00161612 max lat: 0.766251 avg lat: 0.171577 2023-01-25T11:10:58.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:10:58.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 20 16 1862 1846 5.768 5.375 0.201753 0.171577 2023-01-25T11:10:59.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 21 16 1963 1947 5.79388 6.3125 0.104598 0.171787 2023-01-25T11:11:00.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 22 16 2049 2033 5.77482 5.375 0.00194633 0.171688 2023-01-25T11:11:01.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 23 16 2136 2120 5.76012 5.4375 0.102385 0.172652 2023-01-25T11:11:02.171 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 24 16 2225 2209 5.75187 5.5625 0.102134 0.172752 2023-01-25T11:11:03.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 25 16 2312 2296 5.73927 5.4375 0.00991826 0.173108 2023-01-25T11:11:04.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 26 16 2406 2390 5.74445 5.875 0.00421362 0.172783 2023-01-25T11:11:05.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 27 16 2495 2479 5.73768 5.5625 0.103856 0.173519 2023-01-25T11:11:06.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 28 16 2592 2576 5.74925 6.0625 0.0105129 0.173101 2023-01-25T11:11:07.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 29 16 2682 2666 5.74493 5.625 0.109264 0.173031 2023-01-25T11:11:08.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 30 16 2756 2740 5.70759 4.625 0.00188219 0.172288 2023-01-25T11:11:09.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 31 16 2820 2804 5.6525 4 0.00590909 0.176023 2023-01-25T11:11:10.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 32 16 2921 2905 5.67309 6.3125 0.103789 0.175609 2023-01-25T11:11:11.172 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 33 16 3013 2997 5.67541 5.75 0.00418848 0.175473 2023-01-25T11:11:12.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 34 16 3102 3086 5.67206 5.5625 0.208239 0.17536 2023-01-25T11:11:13.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 35 16 3202 3186 5.68855 6.25 0.298012 0.175525 2023-01-25T11:11:14.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 36 16 3290 3274 5.68328 5.5 0.10198 0.174912 2023-01-25T11:11:15.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 37 16 3391 3375 5.70027 6.3125 0.00290496 0.174965 2023-01-25T11:11:16.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 38 16 3505 3489 5.73773 7.125 0.00852438 0.173743 2023-01-25T11:11:16.957 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:11:16.954+0000 7fa985af6700 -1 osd.5 60 handle_osd_map injecting map crc failure 2023-01-25T11:11:17.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 39 16 3603 3587 5.74764 6.125 0.101782 0.173367 2023-01-25T11:11:17.968 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:11:17.965+0000 7ffad630c700 -1 osd.4 61 handle_osd_map injecting map crc failure 2023-01-25T11:11:18.173 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:11:18.172678+0000 min lat: 0.00161612 max lat: 2.43248 avg lat: 0.173743 2023-01-25T11:11:18.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:11:18.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 40 16 3682 3666 5.72737 4.9375 0.00235455 0.173743 2023-01-25T11:11:19.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 41 16 3764 3748 5.71266 5.125 0.00200462 0.174233 2023-01-25T11:11:20.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 42 16 3855 3839 5.71204 5.6875 0.101998 0.174053 2023-01-25T11:11:21.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 43 16 3951 3935 5.71872 6 0.00193742 0.174256 2023-01-25T11:11:22.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 44 16 4055 4039 5.73645 6.5 0.104311 0.173886 2023-01-25T11:11:23.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 45 16 4158 4142 5.75201 6.4375 0.00700337 0.17356 2023-01-25T11:11:24.174 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 46 16 4255 4239 5.75875 6.0625 0.292617 0.17316 2023-01-25T11:11:25.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 47 16 4346 4330 5.75722 5.6875 0.208444 0.17311 2023-01-25T11:11:26.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 48 16 4433 4417 5.75055 5.4375 0.00217322 0.173426 2023-01-25T11:11:27.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 49 16 4523 4507 5.74798 5.625 0.100705 0.173403 2023-01-25T11:11:28.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 50 16 4614 4598 5.74675 5.6875 0.00256989 0.173584 2023-01-25T11:11:29.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 51 16 4730 4714 5.77621 7.25 0.113285 0.172868 2023-01-25T11:11:30.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 52 16 4849 4833 5.80813 7.4375 0.216104 0.171721 2023-01-25T11:11:31.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 53 16 4939 4923 5.80466 5.625 0.20575 0.171947 2023-01-25T11:11:32.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 54 16 5042 5026 5.81637 6.4375 0.103287 0.171621 2023-01-25T11:11:33.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 55 16 5150 5134 5.83333 6.75 0.00953354 0.171137 2023-01-25T11:11:34.175 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 56 16 5238 5222 5.82737 5.5 0.203157 0.171311 2023-01-25T11:11:35.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 57 16 5337 5321 5.83367 6.1875 0.106846 0.170897 2023-01-25T11:11:36.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 58 16 5434 5418 5.83761 6.0625 0.00194895 0.170855 2023-01-25T11:11:37.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 59 16 5538 5522 5.84882 6.5 0.00375353 0.170669 2023-01-25T11:11:38.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:11:38.175149+0000 min lat: 0.00161612 max lat: 2.43248 avg lat: 0.171137 2023-01-25T11:11:38.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:11:38.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 60 16 5621 5605 5.83779 5.1875 0.00982705 0.171137 2023-01-25T11:11:39.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 61 16 5718 5702 5.84146 6.0625 0.00646664 0.170939 2023-01-25T11:11:40.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 62 16 5804 5788 5.83393 5.375 0.502953 0.171139 2023-01-25T11:11:41.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 63 16 5893 5877 5.8296 5.5625 0.00209678 0.171031 2023-01-25T11:11:42.176 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 64 16 5963 5947 5.80687 4.375 0.102688 0.171822 2023-01-25T11:11:43.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 65 16 6069 6053 5.81944 6.625 0.00698551 0.17161 2023-01-25T11:11:44.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 66 16 6160 6144 5.81743 5.6875 0.00224349 0.171372 2023-01-25T11:11:45.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 67 16 6251 6235 5.81548 5.6875 0.00269845 0.171665 2023-01-25T11:11:46.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 68 16 6343 6327 5.8145 5.75 0.0072801 0.171661 2023-01-25T11:11:47.177 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 69 16 6438 6422 5.81627 5.9375 0.103619 0.171592 2023-01-25T11:11:48.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 70 16 6528 6512 5.81353 5.625 0.303326 0.17173 2023-01-25T11:11:49.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 71 16 6629 6613 5.82055 6.3125 0.00688591 0.171448 2023-01-25T11:11:50.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 72 16 6728 6712 5.82563 6.1875 0.104941 0.171472 2023-01-25T11:11:51.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 73 16 6823 6807 5.82715 5.9375 0.002395 0.171299 2023-01-25T11:11:52.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 74 16 6918 6902 5.82863 5.9375 0.108471 0.17135 2023-01-25T11:11:53.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 75 16 7015 6999 5.83174 6.0625 0.00188963 0.171157 2023-01-25T11:11:54.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 76 16 7136 7120 5.8545 7.5625 0.102264 0.170501 2023-01-25T11:11:55.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 77 16 7236 7220 5.85962 6.25 0.00214737 0.170361 2023-01-25T11:11:56.178 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 78 16 7319 7303 5.851 5.1875 0.00246352 0.170544 2023-01-25T11:11:57.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 79 16 7436 7420 5.86949 7.3125 0.120413 0.170189 2023-01-25T11:11:58.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:11:58.177788+0000 min lat: 0.00161612 max lat: 2.43248 avg lat: 0.170035 2023-01-25T11:11:58.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:11:58.180 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 80 16 7531 7515 5.87033 5.9375 0.0362185 0.170035 2023-01-25T11:11:59.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 81 16 7608 7592 5.85726 4.8125 0.00343208 0.170297 2023-01-25T11:12:00.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 82 16 7685 7669 5.84451 4.8125 0.00183745 0.170776 2023-01-25T11:12:01.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 83 16 7784 7768 5.84864 6.1875 0.105146 0.170721 2023-01-25T11:12:02.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 84 16 7882 7866 5.85192 6.125 0.00649575 0.170684 2023-01-25T11:12:03.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 85 16 7983 7967 5.85733 6.3125 0.207245 0.170536 2023-01-25T11:12:04.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 86 16 8076 8060 5.8568 5.8125 0.297989 0.170466 2023-01-25T11:12:05.179 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 87 16 8170 8154 5.857 5.875 0.0048398 0.170526 2023-01-25T11:12:06.073 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:12:06.070+0000 7ffad630c700 -1 osd.4 62 handle_osd_map injecting map crc failure 2023-01-25T11:12:06.180 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 88 16 8289 8273 5.87495 7.4375 0.274603 0.170035 2023-01-25T11:12:07.180 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 89 16 8372 8356 5.86722 5.1875 0.00800103 0.169613 2023-01-25T11:12:08.180 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 90 16 8444 8428 5.85202 4.5 0.00711834 0.170356 2023-01-25T11:12:09.180 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total time run: 90.2038 2023-01-25T11:12:09.180 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total writes made: 8444 2023-01-25T11:12:09.181 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Write size: 65536 2023-01-25T11:12:09.181 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object size: 65536 2023-01-25T11:12:09.181 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Bandwidth (MB/sec): 5.85064 2023-01-25T11:12:09.181 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Bandwidth: 0.713614 2023-01-25T11:12:09.181 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max bandwidth (MB/sec): 7.5625 2023-01-25T11:12:09.181 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min bandwidth (MB/sec): 4 2023-01-25T11:12:09.182 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average IOPS: 93 2023-01-25T11:12:09.182 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev IOPS: 11.4178 2023-01-25T11:12:09.182 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max IOPS: 121 2023-01-25T11:12:09.182 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min IOPS: 64 2023-01-25T11:12:09.182 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average Latency(s): 0.170819 2023-01-25T11:12:09.183 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Latency(s): 0.138808 2023-01-25T11:12:09.183 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max latency(s): 2.43248 2023-01-25T11:12:09.183 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min latency(s): 0.00161612 2023-01-25T11:12:09.387 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Cleaning up (deleting benchmark objects) 2023-01-25T11:13:37.352 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Removed 8444 objects 2023-01-25T11:13:37.353 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Clean up completed and total clean up time :87.9654 2023-01-25T11:13:41.155 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_3 2023-01-25T11:13:41.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2023-01-25T11:13:41.959 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_3' removed 2023-01-25T11:13:41.972 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2023-01-25T11:13:41.973 INFO:tasks.radosbench:Beginning radosbench... 2023-01-25T11:13:41.973 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_4 2023-01-25T11:13:41.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_4 16 2023-01-25T11:13:42.295 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:13:42.293+0000 7f4d2d1da700 -1 osd.6 66 handle_osd_map injecting map crc failure 2023-01-25T11:13:42.977 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_4' created 2023-01-25T11:13:42.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2023-01-25T11:13:43.966 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:13:43.964+0000 7fcc32ee8700 -1 osd.3 68 handle_osd_map injecting map crc failure 2023-01-25T11:13:43.967 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:13:43.965+0000 7ffad8310700 -1 osd.4 68 handle_osd_map injecting map crc failure 2023-01-25T11:13:43.969 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:13:43.968+0000 7fa987afa700 -1 osd.5 68 handle_osd_map injecting map crc failure 2023-01-25T11:13:43.970 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:13:43.968+0000 7fb8a2941700 -1 osd.7 68 handle_osd_map injecting map crc failure 2023-01-25T11:13:43.974 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_4' 2023-01-25T11:13:44.991 DEBUG:teuthology.orchestra.run.smithi027:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2023-01-25T11:13:44.994 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2023-01-25T11:13:44.994 INFO:teuthology.orchestra.run:waiting for 3000 2023-01-25T11:13:45.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:hints = 1 2023-01-25T11:13:45.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2023-01-25T11:13:45.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object prefix: benchmark_data_smithi027_112753 2023-01-25T11:13:45.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:13:45.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 0 0 0 0 0 0 - 0 2023-01-25T11:13:45.058 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:13:45.056+0000 7f4d2f1de700 -1 osd.6 69 handle_osd_map injecting map crc failure 2023-01-25T11:13:45.674 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:13:45.672+0000 7fb8a2941700 -1 osd.7 69 handle_osd_map injecting map crc failure 2023-01-25T11:13:46.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 1 16 104 88 5.49933 5.5 0.102814 0.147909 2023-01-25T11:13:47.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 2 16 203 187 5.84294 6.1875 0.0069771 0.163334 2023-01-25T11:13:48.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 3 16 291 275 5.72846 5.5 0.00210337 0.168378 2023-01-25T11:13:49.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 4 16 383 367 5.73374 5.75 0.207577 0.170554 2023-01-25T11:13:50.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 5 16 473 457 5.71189 5.625 0.102219 0.172508 2023-01-25T11:13:51.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 6 16 559 543 5.65565 5.375 0.00198564 0.171413 2023-01-25T11:13:52.056 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 7 16 667 651 5.81184 6.75 0.107912 0.168066 2023-01-25T11:13:53.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 8 16 742 726 5.67126 4.6875 0.10446 0.172908 2023-01-25T11:13:54.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 9 16 821 805 5.58964 4.9375 0.304421 0.174997 2023-01-25T11:13:55.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 10 16 932 916 5.72432 6.9375 0.00662049 0.172578 2023-01-25T11:13:56.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 11 16 1018 1002 5.6925 5.375 0.00176566 0.173499 2023-01-25T11:13:57.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 12 16 1129 1113 5.79619 6.9375 0.104391 0.171381 2023-01-25T11:13:58.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 13 16 1216 1200 5.76856 5.4375 0.010321 0.171213 2023-01-25T11:13:59.057 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 14 16 1310 1294 5.7761 5.875 0.102545 0.171387 2023-01-25T11:14:00.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 15 16 1407 1391 5.79513 6.0625 0.0995824 0.171348 2023-01-25T11:14:01.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 16 16 1479 1463 5.71414 4.5 0.102398 0.172922 2023-01-25T11:14:02.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 17 16 1585 1569 5.76766 6.625 0.12225 0.17275 2023-01-25T11:14:03.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 18 16 1665 1649 5.72499 5 0.202097 0.173759 2023-01-25T11:14:04.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 19 16 1743 1727 5.68023 4.875 0.209239 0.175177 2023-01-25T11:14:05.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:14:05.057402+0000 min lat: 0.00145267 max lat: 0.908636 avg lat: 0.175511 2023-01-25T11:14:05.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:14:05.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 20 16 1829 1813 5.66495 5.375 0.10075 0.175511 2023-01-25T11:14:06.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 21 16 1909 1893 5.63326 5 0.1013 0.175573 2023-01-25T11:14:07.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 22 16 2004 1988 5.64706 5.9375 0.104246 0.1761 2023-01-25T11:14:07.091 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:14:07.089+0000 7fa985af6700 -1 osd.5 70 handle_osd_map injecting map crc failure 2023-01-25T11:14:08.058 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 23 16 2090 2074 5.6352 5.375 0.121948 0.174841 2023-01-25T11:14:09.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 24 16 2151 2135 5.55924 3.8125 0.00383211 0.179352 2023-01-25T11:14:10.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 25 16 2260 2244 5.60933 6.8125 0.102617 0.177499 2023-01-25T11:14:11.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 26 16 2345 2329 5.59789 5.3125 0.109404 0.177694 2023-01-25T11:14:12.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 27 16 2439 2423 5.60814 5.875 0.0179352 0.177442 2023-01-25T11:14:13.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 28 16 2534 2518 5.61988 5.9375 0.103132 0.177152 2023-01-25T11:14:14.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 29 16 2626 2610 5.62434 5.75 0.157535 0.177005 2023-01-25T11:14:15.059 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 30 16 2715 2699 5.62225 5.5625 0.00481907 0.177336 2023-01-25T11:14:16.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 31 16 2807 2791 5.62636 5.75 0.00700823 0.177186 2023-01-25T11:14:17.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 32 16 2895 2879 5.62238 5.5 0.00213627 0.176671 2023-01-25T11:14:18.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 33 16 2986 2970 5.62434 5.6875 0.412082 0.177298 2023-01-25T11:14:19.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 34 16 3085 3069 5.64088 6.1875 0.305707 0.17639 2023-01-25T11:14:20.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 35 16 3193 3177 5.67254 6.75 0.101897 0.175542 2023-01-25T11:14:21.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 36 16 3287 3271 5.67814 5.875 0.220427 0.17538 2023-01-25T11:14:22.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 37 16 3390 3374 5.69865 6.4375 0.205928 0.174973 2023-01-25T11:14:23.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 38 16 3484 3468 5.70328 5.875 0.391631 0.174625 2023-01-25T11:14:24.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 39 16 3564 3548 5.68524 5 0.105244 0.175105 2023-01-25T11:14:25.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:14:25.059610+0000 min lat: 0.00145267 max lat: 1.83381 avg lat: 0.175055 2023-01-25T11:14:25.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:14:25.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 40 16 3650 3634 5.67747 5.375 0.00842038 0.175055 2023-01-25T11:14:26.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 41 16 3740 3724 5.67618 5.625 0.1075 0.175527 2023-01-25T11:14:27.060 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 42 16 3828 3812 5.67197 5.5 0.304273 0.175776 2023-01-25T11:14:28.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 43 16 3901 3885 5.64615 4.5625 0.0022515 0.176728 2023-01-25T11:14:29.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 44 16 4006 3990 5.66696 6.5625 0.101696 0.176201 2023-01-25T11:14:30.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 45 16 4096 4080 5.66602 5.625 0.20268 0.176003 2023-01-25T11:14:31.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 46 16 4186 4170 5.66511 5.625 0.103406 0.176015 2023-01-25T11:14:32.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 47 16 4285 4269 5.6762 6.1875 0.00428039 0.175572 2023-01-25T11:14:33.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 48 16 4394 4378 5.69986 6.8125 0.00765809 0.175106 2023-01-25T11:14:34.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 49 16 4498 4482 5.71617 6.5 0.00620934 0.174528 2023-01-25T11:14:35.061 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 50 16 4593 4577 5.72059 5.9375 0.102352 0.174307 2023-01-25T11:14:36.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 51 16 4680 4664 5.71503 5.4375 0.202612 0.174211 2023-01-25T11:14:37.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 52 16 4770 4754 5.71329 5.625 0.0138151 0.174472 2023-01-25T11:14:38.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 53 16 4874 4858 5.72812 6.5 0.112183 0.174204 2023-01-25T11:14:39.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 54 16 4965 4949 5.72735 5.6875 0.00282862 0.174292 2023-01-25T11:14:40.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 55 16 5063 5047 5.73457 6.125 0.00298488 0.174151 2023-01-25T11:14:41.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 56 16 5150 5134 5.72926 5.4375 0.301067 0.173992 2023-01-25T11:14:42.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 57 16 5249 5233 5.73729 6.1875 0.102379 0.173756 2023-01-25T11:14:43.062 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 58 16 5356 5340 5.75365 6.6875 0.202515 0.173501 2023-01-25T11:14:44.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 59 16 5451 5435 5.75676 5.9375 0.102185 0.173294 2023-01-25T11:14:45.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:14:45.061896+0000 min lat: 0.00145267 max lat: 1.83381 avg lat: 0.173243 2023-01-25T11:14:45.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:14:45.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 60 16 5546 5530 5.75976 5.9375 0.0107322 0.173243 2023-01-25T11:14:46.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 61 16 5616 5600 5.73704 4.375 0.101295 0.173969 2023-01-25T11:14:47.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 62 16 5708 5692 5.73725 5.75 0.199276 0.173944 2023-01-25T11:14:48.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 63 16 5807 5791 5.74438 6.1875 0.10912 0.173624 2023-01-25T11:14:49.067 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 64 16 5914 5898 5.75911 6.6875 0.106264 0.173352 2023-01-25T11:14:50.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 65 16 6006 5990 5.75896 5.75 0.09507 0.173205 2023-01-25T11:14:51.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 66 16 6098 6082 5.75881 5.75 0.202522 0.173175 2023-01-25T11:14:52.063 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 67 16 6203 6187 5.77079 6.5625 0.00755535 0.172927 2023-01-25T11:14:53.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 68 16 6299 6283 5.77415 6 0.104886 0.172868 2023-01-25T11:14:54.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 69 16 6408 6392 5.78919 6.8125 0.00536926 0.17254 2023-01-25T11:14:55.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 70 16 6509 6493 5.79665 6.3125 0.10596 0.172055 2023-01-25T11:14:56.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 71 16 6611 6595 5.80479 6.375 0.102554 0.171954 2023-01-25T11:14:57.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 72 16 6690 6674 5.79273 4.9375 0.388097 0.172203 2023-01-25T11:14:58.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 73 16 6789 6773 5.79813 6.1875 0.101716 0.172105 2023-01-25T11:14:59.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 74 16 6893 6877 5.8076 6.5 0.0998427 0.171873 2023-01-25T11:15:00.064 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 75 16 7009 6993 5.82682 7.25 0.102221 0.171279 2023-01-25T11:15:01.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 76 16 7123 7107 5.84389 7.125 0.00203846 0.170917 2023-01-25T11:15:02.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 77 16 7224 7208 5.84996 6.3125 0.208842 0.170565 2023-01-25T11:15:03.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 78 16 7328 7312 5.85828 6.5 0.00418778 0.170497 2023-01-25T11:15:04.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 79 16 7419 7403 5.85611 5.6875 0.00875527 0.170512 2023-01-25T11:15:05.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:2023-01-25T11:15:05.064475+0000 min lat: 0.00145267 max lat: 1.83381 avg lat: 0.170817 2023-01-25T11:15:05.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-01-25T11:15:05.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 80 16 7501 7485 5.84696 5.125 0.0201143 0.170817 2023-01-25T11:15:06.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 81 16 7604 7588 5.85424 6.4375 0.204315 0.170511 2023-01-25T11:15:07.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 82 16 7696 7680 5.85297 5.75 0.307797 0.170666 2023-01-25T11:15:07.964 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:15:07.961+0000 7f5f3359d700 -1 osd.0 73 handle_osd_map injecting map crc failure 2023-01-25T11:15:08.065 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 83 16 7778 7762 5.84419 5.125 0.205569 0.170865 2023-01-25T11:15:08.929 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-01-25T11:15:08.927+0000 7fcc32ee8700 -1 osd.3 74 handle_osd_map injecting map crc failure 2023-01-25T11:15:08.932 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-01-25T11:15:08.930+0000 7f5f355a1700 -1 osd.0 74 handle_osd_map injecting map crc failure 2023-01-25T11:15:08.971 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-01-25T11:15:08.968+0000 7fac7fb01700 -1 osd.1 74 handle_osd_map injecting map crc failure 2023-01-25T11:15:09.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 84 16 7879 7863 5.84976 6.3125 0.272566 0.170726 2023-01-25T11:15:10.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 85 16 7974 7958 5.85078 5.9375 0.10423 0.170464 2023-01-25T11:15:10.192 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:15:10.190+0000 7ffad630c700 -1 osd.4 75 handle_osd_map injecting map crc failure 2023-01-25T11:15:10.293 INFO:tasks.ceph.osd.5.smithi080.stderr:2023-01-25T11:15:10.291+0000 7fa985af6700 -1 osd.5 75 handle_osd_map injecting map crc failure 2023-01-25T11:15:11.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 86 16 8055 8039 5.8416 5.0625 0.302431 0.170718 2023-01-25T11:15:12.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 87 16 8157 8141 5.84773 6.375 0.111579 0.170866 2023-01-25T11:15:13.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 88 16 8269 8253 5.86081 7 0.0121037 0.170496 2023-01-25T11:15:14.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 89 16 8353 8337 5.85394 5.25 0.10423 0.170502 2023-01-25T11:15:15.066 INFO:tasks.radosbench.radosbench.0.smithi027.stdout: 90 16 8449 8433 5.85556 6 0.103351 0.170474 2023-01-25T11:15:16.067 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total time run: 90.2214 2023-01-25T11:15:16.067 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Total writes made: 8449 2023-01-25T11:15:16.067 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Write size: 65536 2023-01-25T11:15:16.068 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Object size: 65536 2023-01-25T11:15:16.068 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Bandwidth (MB/sec): 5.85296 2023-01-25T11:15:16.068 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Bandwidth: 0.670693 2023-01-25T11:15:16.068 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max bandwidth (MB/sec): 7.25 2023-01-25T11:15:16.068 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min bandwidth (MB/sec): 3.8125 2023-01-25T11:15:16.069 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average IOPS: 93 2023-01-25T11:15:16.069 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev IOPS: 10.7311 2023-01-25T11:15:16.069 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max IOPS: 116 2023-01-25T11:15:16.069 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min IOPS: 61 2023-01-25T11:15:16.069 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Average Latency(s): 0.170605 2023-01-25T11:15:16.070 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Stddev Latency(s): 0.131941 2023-01-25T11:15:16.070 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Max latency(s): 1.83381 2023-01-25T11:15:16.070 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Min latency(s): 0.00145267 2023-01-25T11:15:16.273 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Cleaning up (deleting benchmark objects) 2023-01-25T11:16:08.102 INFO:tasks.ceph.osd.4.smithi080.stderr:2023-01-25T11:16:08.100+0000 7ffad630c700 -1 osd.4 77 handle_osd_map injecting map crc failure 2023-01-25T11:16:09.025 INFO:tasks.ceph.osd.6.smithi080.stderr:2023-01-25T11:16:09.023+0000 7f4d2d1da700 -1 osd.6 78 handle_osd_map injecting map crc failure 2023-01-25T11:16:45.094 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Removed 8449 objects 2023-01-25T11:16:45.094 INFO:tasks.radosbench.radosbench.0.smithi027.stdout:Clean up completed and total clean up time :88.8202 2023-01-25T11:16:48.038 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_4 2023-01-25T11:16:48.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2023-01-25T11:16:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_4' removed 2023-01-25T11:16:49.107 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-25T11:16:49.117 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-25T11:16:49.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T11:16:49.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:16:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-01-25T11:16:49.469 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2023-01-25T11:16:49.085253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267872,"kb_used_data":10912,"kb_used_omap":0,"kb_used_meta":256896,"kb_avail":754706848,"statfs":{"total":773094113280,"available":772819812352,"internally_reserved":0,"allocated":11173888,"data_stored":7369360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":263061504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-56098840,"num_objects":-857,"num_object_clones":0,"num_object_copies":-1714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-857,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-16043,"num_write_kb":-540737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-125,"num_bytes_recovered":-8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.260396"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.117986+0000","last_change":"2023-01-25T11:00:55.666575+0000","last_active":"2023-01-25T11:16:08.117986+0000","last_peered":"2023-01-25T11:16:08.117986+0000","last_clean":"2023-01-25T11:16:08.117986+0000","last_became_active":"2023-01-25T11:00:55.665925+0000","last_became_peered":"2023-01-25T11:00:55.665925+0000","last_unstale":"2023-01-25T11:16:08.117986+0000","last_undegraded":"2023-01-25T11:16:08.117986+0000","last_fullsized":"2023-01-25T11:16:08.117986+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T18:15:28.854851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.103262+0000","last_change":"2023-01-25T11:00:55.662604+0000","last_active":"2023-01-25T11:16:08.103262+0000","last_peered":"2023-01-25T11:16:08.103262+0000","last_clean":"2023-01-25T11:16:08.103262+0000","last_became_active":"2023-01-25T11:00:55.662379+0000","last_became_peered":"2023-01-25T11:00:55.662379+0000","last_unstale":"2023-01-25T11:16:08.103262+0000","last_undegraded":"2023-01-25T11:16:08.103262+0000","last_fullsized":"2023-01-25T11:16:08.103262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T16:06:28.820568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-01-25T11:16:09.027927+0000","last_change":"2023-01-25T11:16:09.027927+0000","last_active":"2023-01-25T11:16:09.027927+0000","last_peered":"2023-01-25T11:16:09.027927+0000","last_clean":"2023-01-25T11:16:09.027927+0000","last_became_active":"2023-01-25T11:16:09.027670+0000","last_became_peered":"2023-01-25T11:16:09.027670+0000","last_unstale":"2023-01-25T11:16:09.027927+0000","last_undegraded":"2023-01-25T11:16:09.027927+0000","last_fullsized":"2023-01-25T11:16:09.027927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:10:57.512638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.103299+0000","last_change":"2023-01-25T11:00:55.659957+0000","last_active":"2023-01-25T11:16:08.103299+0000","last_peered":"2023-01-25T11:16:08.103299+0000","last_clean":"2023-01-25T11:16:08.103299+0000","last_became_active":"2023-01-25T11:00:55.659687+0000","last_became_peered":"2023-01-25T11:00:55.659687+0000","last_unstale":"2023-01-25T11:16:08.103299+0000","last_undegraded":"2023-01-25T11:16:08.103299+0000","last_fullsized":"2023-01-25T11:16:08.103299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T22:07:34.953392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":199,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.020114+0000","last_change":"2023-01-25T11:15:11.244549+0000","last_active":"2023-01-25T11:16:08.020114+0000","last_peered":"2023-01-25T11:16:08.020114+0000","last_clean":"2023-01-25T11:16:08.020114+0000","last_became_active":"2023-01-25T11:15:11.244239+0000","last_became_peered":"2023-01-25T11:15:11.244239+0000","last_unstale":"2023-01-25T11:16:08.020114+0000","last_undegraded":"2023-01-25T11:16:08.020114+0000","last_fullsized":"2023-01-25T11:16:08.020114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_clean_scrub_stamp":"2023-01-25T11:00:53.636416+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T22:46:43.914511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.020121+0000","last_change":"2023-01-25T11:00:55.891233+0000","last_active":"2023-01-25T11:16:08.020121+0000","last_peered":"2023-01-25T11:16:08.020121+0000","last_clean":"2023-01-25T11:16:08.020121+0000","last_became_active":"2023-01-25T11:00:55.890849+0000","last_became_peered":"2023-01-25T11:00:55.890849+0000","last_unstale":"2023-01-25T11:16:08.020121+0000","last_undegraded":"2023-01-25T11:16:08.020121+0000","last_fullsized":"2023-01-25T11:16:08.020121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:22:26.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.020220+0000","last_change":"2023-01-25T11:00:55.666198+0000","last_active":"2023-01-25T11:16:08.020220+0000","last_peered":"2023-01-25T11:16:08.020220+0000","last_clean":"2023-01-25T11:16:08.020220+0000","last_became_active":"2023-01-25T11:00:55.665948+0000","last_became_peered":"2023-01-25T11:00:55.665948+0000","last_unstale":"2023-01-25T11:16:08.020220+0000","last_undegraded":"2023-01-25T11:16:08.020220+0000","last_fullsized":"2023-01-25T11:16:08.020220+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:29:29.559334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.021525+0000","last_change":"2023-01-25T11:04:05.139013+0000","last_active":"2023-01-25T11:16:08.021525+0000","last_peered":"2023-01-25T11:16:08.021525+0000","last_clean":"2023-01-25T11:16:08.021525+0000","last_became_active":"2023-01-25T11:04:05.138728+0000","last_became_peered":"2023-01-25T11:04:05.138728+0000","last_unstale":"2023-01-25T11:16:08.021525+0000","last_undegraded":"2023-01-25T11:16:08.021525+0000","last_fullsized":"2023-01-25T11:16:08.021525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T11:35:48.790743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":79,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-01-25T11:16:09.125862+0000","last_change":"2023-01-25T11:00:55.750203+0000","last_active":"2023-01-25T11:16:09.125862+0000","last_peered":"2023-01-25T11:16:09.125862+0000","last_clean":"2023-01-25T11:16:09.125862+0000","last_became_active":"2023-01-25T11:00:55.749892+0000","last_became_peered":"2023-01-25T11:00:55.749892+0000","last_unstale":"2023-01-25T11:16:09.125862+0000","last_undegraded":"2023-01-25T11:16:09.125862+0000","last_fullsized":"2023-01-25T11:16:09.125862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:09:11.429769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37808,"kb_used_data":1320,"kb_used_omap":0,"kb_used_meta":36480,"kb_avail":94334032,"statfs":{"total":96636764160,"available":96598048768,"internally_reserved":0,"allocated":1351680,"data_stored":882457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37355520},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":33.037999999999997,"5min":22.603999999999999,"15min":14.319000000000001},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.22800000000000001},"max":{"1min":362.88499999999999,"5min":362.88499999999999,"15min":362.88499999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":16.782,"5min":19.846,"15min":15.276999999999999},"min":{"1min":0.40500000000000003,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":87.055000000000007,"5min":231.05799999999999,"15min":250.303},"last":0.38100000000000001}]},{"osd":1,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":17.891999999999999,"5min":21.568999999999999,"15min":19.986000000000001},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.27300000000000002},"max":{"1min":87.159000000000006,"5min":231.09299999999999,"15min":231.09299999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":4.3129999999999997,"5min":7.0860000000000003,"15min":11.462},"min":{"1min":0.36799999999999999,"5min":0.28599999999999998,"15min":0.255},"max":{"1min":80.888000000000005,"5min":166.459,"15min":166.459},"last":0.496}]},{"osd":2,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":15.912000000000001,"5min":12.365,"15min":8.4480000000000004},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.26600000000000001},"max":{"1min":104.443,"5min":104.443,"15min":163.81700000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":18.402000000000001,"5min":16.648,"15min":21},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.308},"max":{"1min":87.117999999999995,"5min":231.113,"15min":318.10599999999999},"last":0.48399999999999999}]},{"osd":3,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":9.3000000000000007,"15min":9.1820000000000004},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.28100000000000003},"max":{"1min":0.621,"5min":136.96100000000001,"15min":157.691},"last":0.39700000000000002},{"interface":"front","average":{"1min":27.673999999999999,"5min":26.567,"15min":16.152000000000001},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":178.41999999999999,"5min":293.48000000000002,"15min":293.48000000000002},"last":0.44500000000000001}]},{"osd":4,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":13.125,"5min":9.7409999999999997,"15min":18.032},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.22900000000000001},"max":{"1min":169.56100000000001,"5min":169.56100000000001,"15min":213.69999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":6.306,"5min":18.202999999999999,"15min":14.609999999999999},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":54.042999999999999,"5min":256.87799999999999,"15min":280.24599999999998},"last":0.33000000000000002}]},{"osd":5,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":15.02,"5min":23.286000000000001,"15min":19.399000000000001},"min":{"1min":0.308,"5min":0.28699999999999998,"15min":0.27500000000000002},"max":{"1min":86.936000000000007,"5min":231.13300000000001,"15min":231.13300000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.486,"5min":8.4870000000000001,"15min":10.06},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":22.335000000000001,"5min":145.10400000000001,"15min":162.10599999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":3.218,"5min":6.8760000000000003,"15min":8.1980000000000004},"min":{"1min":0.32500000000000001,"5min":0.28399999999999997,"15min":0.27800000000000002},"max":{"1min":58.448999999999998,"5min":153.49100000000001,"15min":153.49100000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.2269999999999999,"5min":9.7449999999999992,"15min":6.9320000000000004},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":58.460999999999999,"5min":153.45699999999999,"15min":153.45699999999999},"last":0.42199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25008,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94346832,"statfs":{"total":96636764160,"available":96611155968,"internally_reserved":0,"allocated":827392,"data_stored":354874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":27.518999999999998,"5min":21.922000000000001,"15min":21.248999999999999},"min":{"1min":0.36499999999999999,"5min":0.27300000000000002,"15min":0.21299999999999999},"max":{"1min":146.20099999999999,"5min":254.50299999999999,"15min":339.80599999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":26.100000000000001,"5min":14.125,"15min":13.346},"min":{"1min":0.35499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":281.52600000000001,"5min":281.52600000000001,"15min":281.52600000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":17.318000000000001,"5min":11.913,"15min":15.593},"min":{"1min":0.34399999999999997,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":183.36699999999999,"5min":183.36699999999999,"15min":256.67200000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":10.452999999999999,"5min":10.468,"15min":8.8740000000000006},"min":{"1min":0.378,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":85.468999999999994,"5min":114.57299999999999,"15min":195.858},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Jan 25 11:16:10 2023","interfaces":[{"interface":"back","average":{"1min":4.8019999999999996,"5min":6.048,"15min":6.9039999999999999},"min":{"1min":0.41099999999999998,"5min":0.34000000000000002,"15min":0.253},"max":{"1min":54.823,"5min":107.956,"15min":195.90899999999999},"last":0.626},{"interface":"front","average":{"1min":55.298000000000002,"5min":20.795000000000002,"15min":18.675999999999998},"min":{"1min":0.33100000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":377.28500000000003,"5min":377.28500000000003,"15min":377.28500000000003},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":58.204000000000001,"5min":34.033000000000001,"15min":22.539000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":316.32400000000001,"5min":329.22399999999999,"15min":329.22399999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":19.952999999999999,"5min":16.109999999999999,"15min":15.555},"min":{"1min":0.377,"5min":0.29599999999999999,"15min":0.26700000000000002},"max":{"1min":183.39599999999999,"5min":183.39599999999999,"15min":301.74200000000002},"last":0.63900000000000001}]},{"osd":4,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":16.503,"5min":8.0030000000000001,"15min":8.8759999999999994},"min":{"1min":0.30199999999999999,"5min":0.26000000000000001,"15min":0.222},"max":{"1min":165.761,"5min":165.761,"15min":195.93899999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":36.630000000000003,"5min":18.306000000000001,"15min":17.878},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.248},"max":{"1min":323.67200000000003,"5min":323.67200000000003,"15min":323.67200000000003},"last":0.50900000000000001}]},{"osd":5,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":15.914999999999999,"5min":9.298,"15min":14.316000000000001},"min":{"1min":0.27000000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":183.33699999999999,"5min":183.33699999999999,"15min":232.00700000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":7.8140000000000001,"5min":11.016,"15min":11.502000000000001},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":54.789999999999999,"5min":184.25200000000001,"15min":291.60000000000002},"last":0.45900000000000002}]},{"osd":7,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":26.053000000000001,"5min":12.672000000000001,"15min":10.465999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28499999999999998},"max":{"1min":281.62,"5min":281.62,"15min":281.62},"last":0.442},{"interface":"front","average":{"1min":16.683,"5min":20.645,"15min":21.347999999999999},"min":{"1min":0.32000000000000001,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":183.29599999999999,"5min":183.29599999999999,"15min":329.608},"last":0.53500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29748,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94342092,"statfs":{"total":96636764160,"available":96606302208,"internally_reserved":0,"allocated":831488,"data_stored":355155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":17.122,"5min":17.757000000000001,"15min":16.099},"min":{"1min":0.26200000000000001,"5min":0.217,"15min":0.19900000000000001},"max":{"1min":198.08199999999999,"5min":200.315,"15min":200.315},"last":0.46800000000000003},{"interface":"front","average":{"1min":4.3399999999999999,"5min":8.6910000000000007,"15min":14.369999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":50.058,"5min":175.48599999999999,"15min":261.98200000000003},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":7.4100000000000001,"15min":10.912000000000001},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":0.42199999999999999,"5min":181.09899999999999,"15min":181.09899999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":28.797999999999998,"5min":22.984000000000002,"15min":15.314},"min":{"1min":0.29599999999999999,"5min":0.24299999999999999,"15min":0.23000000000000001},"max":{"1min":95.108000000000004,"5min":239.709,"15min":239.709},"last":0.56899999999999995}]},{"osd":3,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":9.6959999999999997,"5min":19.744,"15min":16.367999999999999},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":97.210999999999999,"5min":182.119,"15min":266.98700000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":2.754,"5min":11.243,"15min":17.622},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":50.094999999999999,"5min":252.94399999999999,"15min":294.005},"last":0.59099999999999997}]},{"osd":4,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":6.2649999999999997,"15min":13.388999999999999},"min":{"1min":0.37,"5min":0.32200000000000001,"15min":0.29899999999999999},"max":{"1min":59.338999999999999,"5min":100.40000000000001,"15min":259.99099999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":4.4020000000000001,"5min":18.411999999999999,"15min":12.458},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":80.903999999999996,"5min":212.464,"15min":212.464},"last":0.68400000000000005}]},{"osd":5,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":10.648999999999999,"15min":16.501000000000001},"min":{"1min":0.39700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":50.256999999999998,"5min":146.64699999999999,"15min":262.11799999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":4.7089999999999996,"15min":13.087},"min":{"1min":0.42999999999999999,"5min":0.35899999999999999,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":100.36,"15min":177.58799999999999},"last":0.753}]},{"osd":6,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":12.359,"15min":7.0919999999999996},"min":{"1min":0.45200000000000001,"5min":0.39300000000000002,"15min":0.34100000000000003},"max":{"1min":0.751,"5min":181.80600000000001,"15min":181.80600000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":19.236000000000001,"5min":14.813000000000001,"15min":16.228000000000002},"min":{"1min":0.46999999999999997,"5min":0.38800000000000001,"15min":0.378},"max":{"1min":235.804,"5min":249.042,"15min":479.07799999999997},"last":0.76800000000000002}]},{"osd":7,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":7.3630000000000004,"15min":13.981},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.37},"max":{"1min":50.210999999999999,"5min":146.613,"15min":275.94400000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":6.9089999999999998,"5min":21.227,"15min":19.664000000000001},"min":{"1min":0.46100000000000002,"5min":0.435,"15min":0.38700000000000001},"max":{"1min":86.629000000000005,"5min":346.58199999999999,"15min":346.58199999999999},"last":0.69799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575039,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":4708,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":4820992,"data_stored":4353333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":24.588999999999999,"5min":13.138999999999999,"15min":11.493},"min":{"1min":0.22700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":251.64400000000001,"5min":251.64400000000001,"15min":284.37},"last":0.42299999999999999},{"interface":"front","average":{"1min":5.8390000000000004,"5min":8.8870000000000005,"15min":10.361000000000001},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.219},"max":{"1min":98.867999999999995,"5min":173.863,"15min":187.32599999999999},"last":0.497}]},{"osd":2,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":7.2789999999999999,"15min":12.754},"min":{"1min":0.316,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":17.696999999999999,"5min":190.631,"15min":284.41399999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":16.431000000000001,"5min":16.465,"15min":11.597},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":132.00899999999999,"5min":244.227,"15min":244.227},"last":0.61399999999999999}]},{"osd":3,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":19.317,"5min":23.048999999999999,"15min":21.265999999999998},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.246},"max":{"1min":90.551000000000002,"5min":290.13,"15min":290.13},"last":0.52900000000000003},{"interface":"front","average":{"1min":14.680999999999999,"5min":20.076000000000001,"15min":19.091999999999999},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.251},"max":{"1min":145.03399999999999,"5min":212.107,"15min":284.43700000000001},"last":0.57199999999999995}]},{"osd":4,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":27.614000000000001,"5min":19.491,"15min":23.684999999999999},"min":{"1min":0.38800000000000001,"5min":0.35099999999999998,"15min":0.28799999999999998},"max":{"1min":90.620999999999995,"5min":207.87299999999999,"15min":275.28699999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":3.0449999999999999,"5min":10.977,"15min":15.218},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.33200000000000002},"max":{"1min":45.966000000000001,"5min":190.65000000000001,"15min":284.63999999999999},"last":0.65000000000000002}]},{"osd":5,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":6.0389999999999997,"5min":7.633,"15min":9.5380000000000003},"min":{"1min":0.46100000000000002,"5min":0.34999999999999998,"15min":0.32700000000000001},"max":{"1min":98.953000000000003,"5min":174.005,"15min":203.709},"last":0.68999999999999995},{"interface":"front","average":{"1min":24.591999999999999,"5min":30.425000000000001,"15min":19.57},"min":{"1min":0.42999999999999999,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":153.24299999999999,"5min":207.911,"15min":207.911},"last":0.67300000000000004}]},{"osd":6,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":13.372,"5min":11.397,"15min":11.327},"min":{"1min":0.46999999999999997,"5min":0.36199999999999999,"15min":0.317},"max":{"1min":117.518,"5min":190.703,"15min":284.61599999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":11.795,"5min":13.506,"15min":8.0380000000000003},"min":{"1min":0.45300000000000001,"5min":0.36699999999999999,"15min":0.34899999999999998},"max":{"1min":98.975999999999999,"5min":178.78800000000001,"15min":178.78800000000001},"last":43.134}]},{"osd":7,"last update":"Wed Jan 25 11:16:21 2023","interfaces":[{"interface":"back","average":{"1min":10.031000000000001,"5min":14.58,"15min":11.669},"min":{"1min":0.47999999999999998,"5min":0.36099999999999999,"15min":0.35899999999999999},"max":{"1min":98.994,"5min":183.732,"15min":183.732},"last":0.66200000000000003},{"interface":"front","average":{"1min":40.307000000000002,"5min":24.684999999999999,"15min":17.224},"min":{"1min":0.48399999999999999,"5min":0.434,"15min":0.41099999999999998},"max":{"1min":292.41800000000001,"5min":292.41800000000001,"15min":292.41800000000001},"last":0.63700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38456,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":37632,"kb_avail":94333384,"statfs":{"total":96636764160,"available":96597385216,"internally_reserved":0,"allocated":835584,"data_stored":355739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38535168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":26.033999999999999,"5min":21.315999999999999,"15min":25.721},"min":{"1min":0.22,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":298.42700000000002,"5min":462.94200000000001,"15min":462.94200000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":35.530999999999999,"5min":17.585999999999999,"15min":11.266999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22500000000000001},"max":{"1min":155.64400000000001,"5min":167.27099999999999,"15min":176.23099999999999},"last":72.165000000000006}]},{"osd":1,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":16.132999999999999,"5min":13.568,"15min":23.193999999999999},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26000000000000001},"max":{"1min":195.374,"5min":263.44799999999998,"15min":394.91300000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":25.193999999999999,"5min":16.001000000000001,"15min":15.564},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":129.62700000000001,"5min":167.321,"15min":177.40000000000001},"last":72.212999999999994}]},{"osd":3,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":10.733000000000001,"5min":8.3079999999999998,"15min":5.2380000000000004},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.253},"max":{"1min":90.206999999999994,"5min":90.206999999999994,"15min":225.642},"last":0.34200000000000003},{"interface":"front","average":{"1min":31.655999999999999,"5min":17.603000000000002,"15min":15.457000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28399999999999997},"max":{"1min":195.428,"5min":195.428,"15min":394.738},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":40.116999999999997,"5min":19.306999999999999,"15min":27.832999999999998},"min":{"1min":0.41299999999999998,"5min":0.35999999999999999,"15min":0.34100000000000003},"max":{"1min":423.40300000000002,"5min":423.40300000000002,"15min":423.40300000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":26.638000000000002,"5min":13.327,"15min":15.369},"min":{"1min":0.441,"5min":0.38300000000000001,"15min":0.318},"max":{"1min":141.04900000000001,"5min":167.482,"15min":176.40199999999999},"last":72.25}]},{"osd":5,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":20.797000000000001,"5min":16.363,"15min":16.082999999999998},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.29399999999999998},"max":{"1min":129.684,"5min":224.52099999999999,"15min":279.50999999999999},"last":72.271000000000001},{"interface":"front","average":{"1min":18.082999999999998,"5min":12.051,"15min":7.4630000000000001},"min":{"1min":0.40899999999999997,"5min":0.371,"15min":0.313},"max":{"1min":88.242999999999995,"5min":119.569,"15min":225.78200000000001},"last":0.38900000000000001}]},{"osd":6,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":18.135000000000002,"5min":11.986000000000001,"15min":8.8149999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":92.335999999999999,"5min":175.773,"15min":225.75999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":20.504999999999999,"5min":13.470000000000001,"15min":16.210999999999999},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.38200000000000001},"max":{"1min":195.512,"5min":195.512,"15min":394.80399999999997},"last":0.41199999999999998}]},{"osd":7,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":20.829999999999998,"5min":15.722,"15min":11.731},"min":{"1min":0.47899999999999998,"5min":0.40999999999999998,"15min":0.374},"max":{"1min":129.703,"5min":167.50899999999999,"15min":179.28},"last":72.304000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":2.6970000000000001,"15min":4.5069999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77400000000000002,"5min":83.195999999999998,"15min":225.72900000000001},"last":0.50800000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575043,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35092,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":34240,"kb_avail":94336748,"statfs":{"total":96636764160,"available":96600829952,"internally_reserved":0,"allocated":864256,"data_stored":360507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35061760},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":19.295999999999999,"5min":30.321999999999999,"15min":25.77},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":170.369,"5min":254.995,"15min":277.22399999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":15.510999999999999,"5min":18.419,"15min":8.2270000000000003},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.21099999999999999},"max":{"1min":131.34800000000001,"5min":185.78999999999999,"15min":185.78999999999999},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":10.625999999999999,"15min":16.712},"min":{"1min":0.254,"5min":0.249,"15min":0.21299999999999999},"max":{"1min":0.85899999999999999,"5min":182.786,"15min":200.09700000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":5.3319999999999999,"5min":24.030999999999999,"15min":16.286999999999999},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.23499999999999999},"max":{"1min":60.161999999999999,"5min":240.67500000000001,"15min":342.93200000000002},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":10.066000000000001,"5min":11.792,"15min":20.100000000000001},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.24199999999999999},"max":{"1min":196.54900000000001,"5min":196.54900000000001,"15min":314.65199999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":4.5279999999999996,"5min":19.981999999999999,"15min":13.25},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.25900000000000001},"max":{"1min":60.134999999999998,"5min":240.71000000000001,"15min":240.71000000000001},"last":0.496}]},{"osd":4,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":13.728,"5min":22.001000000000001,"15min":13.416},"min":{"1min":0.46400000000000002,"5min":0.378,"15min":0.28899999999999998},"max":{"1min":131.40199999999999,"5min":196.387,"15min":255.024},"last":0.71199999999999997},{"interface":"front","average":{"1min":9.0350000000000001,"5min":11.247999999999999,"15min":22.387},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.26700000000000002},"max":{"1min":129.00399999999999,"5min":182.827,"15min":247.816},"last":0.65000000000000002}]},{"osd":5,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":5.9850000000000003,"15min":14.800000000000001},"min":{"1min":0.375,"5min":0.35199999999999998,"15min":0.307},"max":{"1min":0.95799999999999996,"5min":182.68000000000001,"15min":276.24099999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":18.594999999999999,"5min":35.615000000000002,"15min":18.843},"min":{"1min":0.40100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":160.209,"5min":319.58100000000002,"15min":319.58100000000002},"last":0.71799999999999997}]},{"osd":6,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":14.561999999999999,"5min":18.238,"15min":9.3580000000000005},"min":{"1min":0.41999999999999998,"5min":0.379,"15min":0.26400000000000001},"max":{"1min":131.46700000000001,"5min":185.62200000000001,"15min":185.62200000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":3.8290000000000002,"5min":14.177,"15min":14.451000000000001},"min":{"1min":0.36799999999999999,"5min":0.33500000000000002,"15min":0.32900000000000001},"max":{"1min":33.692,"5min":199.071,"15min":200.161},"last":0.60199999999999998}]},{"osd":7,"last update":"Wed Jan 25 11:16:28 2023","interfaces":[{"interface":"back","average":{"1min":13.369,"5min":33.887999999999998,"15min":24.07},"min":{"1min":0.379,"5min":0.379,"15min":0.378},"max":{"1min":202.59700000000001,"5min":333.29199999999997,"15min":333.29199999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":6.1550000000000002,"5min":21.129000000000001,"15min":8.7230000000000008},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.28499999999999998},"max":{"1min":131.43600000000001,"5min":260.75999999999999,"15min":260.75999999999999},"last":0.58099999999999996}]}]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34476,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":33664,"kb_avail":94337364,"statfs":{"total":96636764160,"available":96601460736,"internally_reserved":0,"allocated":823296,"data_stored":353788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34471936},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":7.9889999999999999,"5min":18.102,"15min":15.618},"min":{"1min":0.25,"5min":0.25,"15min":0.23799999999999999},"max":{"1min":93.814999999999998,"5min":179.048,"15min":326.68799999999999},"last":0.434},{"interface":"front","average":{"1min":31.292999999999999,"5min":17.864999999999998,"15min":18.084},"min":{"1min":0.38,"5min":0.27900000000000003,"15min":0.248},"max":{"1min":178.62299999999999,"5min":187.81,"15min":206.27000000000001},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":9.0259999999999998,"15min":23.591999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23100000000000001},"max":{"1min":7.3140000000000001,"5min":170.44999999999999,"15min":266.084},"last":0.46100000000000002},{"interface":"front","average":{"1min":14.353,"5min":26.155999999999999,"15min":17.254000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":93.893000000000001,"5min":481.20400000000001,"15min":481.20400000000001},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":4.9640000000000004,"5min":1.3740000000000001,"15min":6.6600000000000001},"min":{"1min":0.30299999999999999,"5min":0.28899999999999998,"15min":0.26300000000000001},"max":{"1min":90.438999999999993,"5min":90.438999999999993,"15min":216.55199999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":25.363,"5min":24.550999999999998,"15min":17.449000000000002},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.27500000000000002},"max":{"1min":175.26300000000001,"5min":179.11099999999999,"15min":200.09299999999999},"last":0.503}]},{"osd":3,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":8.8399999999999999,"5min":7.7050000000000001,"15min":6.6399999999999997},"min":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.27700000000000002},"max":{"1min":90.456999999999994,"5min":99.073999999999998,"15min":194.33500000000001},"last":0.498},{"interface":"front","average":{"1min":3.157,"5min":12.273,"15min":24.597999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.314},"max":{"1min":47.500999999999998,"5min":181.923,"15min":206.56800000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":16.225999999999999,"5min":31.553000000000001,"15min":18.413},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.22900000000000001},"max":{"1min":93.769999999999996,"5min":179.005,"15min":193.524},"last":0.35299999999999998},{"interface":"front","average":{"1min":4.9039999999999999,"5min":1.28,"15min":10.529},"min":{"1min":0.28499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":90.379000000000005,"5min":90.379000000000005,"15min":259.06599999999997},"last":0.33200000000000002}]},{"osd":6,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":10.632999999999999,"15min":22.352},"min":{"1min":0.29099999999999998,"5min":0.28000000000000003,"15min":0.26100000000000001},"max":{"1min":15.082000000000001,"5min":170.404,"15min":256.62700000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":8.7360000000000007,"5min":18.548999999999999,"15min":17.408000000000001},"min":{"1min":0.29999999999999999,"5min":0.28499999999999998,"15min":0.26800000000000002},"max":{"1min":93.930999999999997,"5min":179.285,"15min":225.15000000000001},"last":0.40999999999999998}]},{"osd":7,"last update":"Wed Jan 25 11:16:20 2023","interfaces":[{"interface":"back","average":{"1min":11.346,"5min":13.872999999999999,"15min":16.616},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.248},"max":{"1min":93.619,"5min":170.07900000000001,"15min":300.25599999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":7.1420000000000003,"5min":15.579000000000001,"15min":20.050999999999998},"min":{"1min":0.28999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":93.415000000000006,"5min":267.45699999999999,"15min":267.45699999999999},"last":0.38300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30376,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":29568,"kb_avail":94341464,"statfs":{"total":96636764160,"available":96605659136,"internally_reserved":0,"allocated":819200,"data_stored":353507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30277632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":15.569000000000001,"5min":8.0649999999999995,"15min":13.036},"min":{"1min":0.25900000000000001,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":184.44,"5min":184.44,"15min":356.77699999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":17.460999999999999,"5min":12.157,"15min":10.08},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":184.428,"5min":184.428,"15min":290.70999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":26.302,"5min":20.451000000000001,"15min":11.682},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":147.88399999999999,"5min":291.88400000000001,"15min":291.88400000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":3.113,"5min":6.3529999999999998,"15min":19.834},"min":{"1min":0.34799999999999998,"5min":0.27800000000000002,"15min":0.26100000000000001},"max":{"1min":45.878,"5min":93.572999999999993,"15min":356.74799999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":2.492,"5min":9.2490000000000006,"15min":9.6219999999999999},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":32.512999999999998,"5min":195.85599999999999,"15min":290.52800000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":13.006,"5min":12.903,"15min":9.6080000000000005},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.26600000000000001},"max":{"1min":95.251000000000005,"5min":291.916,"15min":291.916},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":11.007,"5min":11.272,"15min":13.327999999999999},"min":{"1min":0.40600000000000003,"5min":0.311,"15min":0.30599999999999999},"max":{"1min":93.436999999999998,"5min":98.018000000000001,"15min":356.71699999999998},"last":0.629},{"interface":"front","average":{"1min":13.044,"5min":15.577,"15min":10.407},"min":{"1min":0.34200000000000003,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":93.454999999999998,"5min":162.208,"15min":290.63600000000002},"last":0.66900000000000004}]},{"osd":4,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":33.244,"5min":14.288,"15min":13.387},"min":{"1min":0.29799999999999999,"5min":0.24099999999999999,"15min":0.21099999999999999},"max":{"1min":159.23500000000001,"5min":291.80700000000002,"15min":291.80700000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":9.2739999999999991,"5min":3.5329999999999999,"15min":17.213999999999999},"min":{"1min":0.28000000000000003,"5min":0.249,"15min":0.249},"max":{"1min":146.56100000000001,"5min":146.56100000000001,"15min":356.79599999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":14.388,"5min":15.894,"15min":14.365},"min":{"1min":0.27900000000000003,"5min":0.26900000000000002,"15min":0.24199999999999999},"max":{"1min":100.18899999999999,"5min":183.089,"15min":290.66899999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":25.193999999999999,"5min":19.401,"15min":13.914999999999999},"min":{"1min":0.33800000000000002,"5min":0.28799999999999998,"15min":0.23899999999999999},"max":{"1min":100.226,"5min":291.851,"15min":291.851},"last":0.58799999999999997}]},{"osd":7,"last update":"Wed Jan 25 11:16:19 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":7.1890000000000001,"15min":12.026999999999999},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":0.70299999999999996,"5min":147.04300000000001,"15min":356.67399999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.4430000000000001,"5min":6.1769999999999996,"15min":8.6229999999999993},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.25600000000000001},"max":{"1min":32.469999999999999,"5min":162.13300000000001,"15min":290.601},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T11:16:49.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T11:16:49.577 INFO:tasks.ceph.osd.7.smithi080.stderr:2023-01-25T11:16:49.575+0000 7fb8a093d700 -1 osd.7 79 handle_osd_map injecting map crc failure 2023-01-25T11:16:49.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-01-25T11:16:49.800 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-01-25T11:16:49.811 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2023-01-25T11:16:49.085253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267872,"kb_used_data":10912,"kb_used_omap":0,"kb_used_meta":256896,"kb_avail":754706848,"statfs":{"total":773094113280,"available":772819812352,"internally_reserved":0,"allocated":11173888,"data_stored":7369360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":263061504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-56098840,"num_objects":-857,"num_object_clones":0,"num_object_copies":-1714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-857,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-16043,"num_write_kb":-540737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-125,"num_bytes_recovered":-8192000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.260396"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.117986+0000","last_change":"2023-01-25T11:00:55.666575+0000","last_active":"2023-01-25T11:16:08.117986+0000","last_peered":"2023-01-25T11:16:08.117986+0000","last_clean":"2023-01-25T11:16:08.117986+0000","last_became_active":"2023-01-25T11:00:55.665925+0000","last_became_peered":"2023-01-25T11:00:55.665925+0000","last_unstale":"2023-01-25T11:16:08.117986+0000","last_undegraded":"2023-01-25T11:16:08.117986+0000","last_fullsized":"2023-01-25T11:16:08.117986+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T18:15:28.854851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.103262+0000","last_change":"2023-01-25T11:00:55.662604+0000","last_active":"2023-01-25T11:16:08.103262+0000","last_peered":"2023-01-25T11:16:08.103262+0000","last_clean":"2023-01-25T11:16:08.103262+0000","last_became_active":"2023-01-25T11:00:55.662379+0000","last_became_peered":"2023-01-25T11:00:55.662379+0000","last_unstale":"2023-01-25T11:16:08.103262+0000","last_undegraded":"2023-01-25T11:16:08.103262+0000","last_fullsized":"2023-01-25T11:16:08.103262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T16:06:28.820568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-01-25T11:16:09.027927+0000","last_change":"2023-01-25T11:16:09.027927+0000","last_active":"2023-01-25T11:16:09.027927+0000","last_peered":"2023-01-25T11:16:09.027927+0000","last_clean":"2023-01-25T11:16:09.027927+0000","last_became_active":"2023-01-25T11:16:09.027670+0000","last_became_peered":"2023-01-25T11:16:09.027670+0000","last_unstale":"2023-01-25T11:16:09.027927+0000","last_undegraded":"2023-01-25T11:16:09.027927+0000","last_fullsized":"2023-01-25T11:16:09.027927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:10:57.512638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.103299+0000","last_change":"2023-01-25T11:00:55.659957+0000","last_active":"2023-01-25T11:16:08.103299+0000","last_peered":"2023-01-25T11:16:08.103299+0000","last_clean":"2023-01-25T11:16:08.103299+0000","last_became_active":"2023-01-25T11:00:55.659687+0000","last_became_peered":"2023-01-25T11:00:55.659687+0000","last_unstale":"2023-01-25T11:16:08.103299+0000","last_undegraded":"2023-01-25T11:16:08.103299+0000","last_fullsized":"2023-01-25T11:16:08.103299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T22:07:34.953392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":199,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.020114+0000","last_change":"2023-01-25T11:15:11.244549+0000","last_active":"2023-01-25T11:16:08.020114+0000","last_peered":"2023-01-25T11:16:08.020114+0000","last_clean":"2023-01-25T11:16:08.020114+0000","last_became_active":"2023-01-25T11:15:11.244239+0000","last_became_peered":"2023-01-25T11:15:11.244239+0000","last_unstale":"2023-01-25T11:16:08.020114+0000","last_undegraded":"2023-01-25T11:16:08.020114+0000","last_fullsized":"2023-01-25T11:16:08.020114+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:53.636416+0000","last_clean_scrub_stamp":"2023-01-25T11:00:53.636416+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T22:46:43.914511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.020121+0000","last_change":"2023-01-25T11:00:55.891233+0000","last_active":"2023-01-25T11:16:08.020121+0000","last_peered":"2023-01-25T11:16:08.020121+0000","last_clean":"2023-01-25T11:16:08.020121+0000","last_became_active":"2023-01-25T11:00:55.890849+0000","last_became_peered":"2023-01-25T11:00:55.890849+0000","last_unstale":"2023-01-25T11:16:08.020121+0000","last_undegraded":"2023-01-25T11:16:08.020121+0000","last_fullsized":"2023-01-25T11:16:08.020121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:22:26.105847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.020220+0000","last_change":"2023-01-25T11:00:55.666198+0000","last_active":"2023-01-25T11:16:08.020220+0000","last_peered":"2023-01-25T11:16:08.020220+0000","last_clean":"2023-01-25T11:16:08.020220+0000","last_became_active":"2023-01-25T11:00:55.665948+0000","last_became_peered":"2023-01-25T11:00:55.665948+0000","last_unstale":"2023-01-25T11:16:08.020220+0000","last_undegraded":"2023-01-25T11:16:08.020220+0000","last_fullsized":"2023-01-25T11:16:08.020220+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T12:29:29.559334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2023-01-25T11:16:08.021525+0000","last_change":"2023-01-25T11:04:05.139013+0000","last_active":"2023-01-25T11:16:08.021525+0000","last_peered":"2023-01-25T11:16:08.021525+0000","last_clean":"2023-01-25T11:16:08.021525+0000","last_became_active":"2023-01-25T11:04:05.138728+0000","last_became_peered":"2023-01-25T11:04:05.138728+0000","last_unstale":"2023-01-25T11:16:08.021525+0000","last_undegraded":"2023-01-25T11:16:08.021525+0000","last_fullsized":"2023-01-25T11:16:08.021525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T11:35:48.790743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":79,"reported_epoch":79,"state":"active+clean","last_fresh":"2023-01-25T11:16:09.125862+0000","last_change":"2023-01-25T11:00:55.750203+0000","last_active":"2023-01-25T11:16:09.125862+0000","last_peered":"2023-01-25T11:16:09.125862+0000","last_clean":"2023-01-25T11:16:09.125862+0000","last_became_active":"2023-01-25T11:00:55.749892+0000","last_became_peered":"2023-01-25T11:00:55.749892+0000","last_unstale":"2023-01-25T11:16:09.125862+0000","last_undegraded":"2023-01-25T11:16:09.125862+0000","last_fullsized":"2023-01-25T11:16:09.125862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T11:00:54.643441+0000","last_clean_scrub_stamp":"2023-01-25T11:00:54.643441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T19:09:11.429769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575043,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37808,"kb_used_data":1320,"kb_used_omap":0,"kb_used_meta":36480,"kb_avail":94334032,"statfs":{"total":96636764160,"available":96598048768,"internally_reserved":0,"allocated":1351680,"data_stored":882457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37355520},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":33.037999999999997,"5min":22.603999999999999,"15min":14.319000000000001},"min":{"1min":0.27000000000000002,"5min":0.25900000000000001,"15min":0.22800000000000001},"max":{"1min":362.88499999999999,"5min":362.88499999999999,"15min":362.88499999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":16.782,"5min":19.846,"15min":15.276999999999999},"min":{"1min":0.40500000000000003,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":87.055000000000007,"5min":231.05799999999999,"15min":250.303},"last":0.38100000000000001}]},{"osd":1,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":17.891999999999999,"5min":21.568999999999999,"15min":19.986000000000001},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.27300000000000002},"max":{"1min":87.159000000000006,"5min":231.09299999999999,"15min":231.09299999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":4.3129999999999997,"5min":7.0860000000000003,"15min":11.462},"min":{"1min":0.36799999999999999,"5min":0.28599999999999998,"15min":0.255},"max":{"1min":80.888000000000005,"5min":166.459,"15min":166.459},"last":0.496}]},{"osd":2,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":15.912000000000001,"5min":12.365,"15min":8.4480000000000004},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.26600000000000001},"max":{"1min":104.443,"5min":104.443,"15min":163.81700000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":18.402000000000001,"5min":16.648,"15min":21},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.308},"max":{"1min":87.117999999999995,"5min":231.113,"15min":318.10599999999999},"last":0.48399999999999999}]},{"osd":3,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":9.3000000000000007,"15min":9.1820000000000004},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.28100000000000003},"max":{"1min":0.621,"5min":136.96100000000001,"15min":157.691},"last":0.39700000000000002},{"interface":"front","average":{"1min":27.673999999999999,"5min":26.567,"15min":16.152000000000001},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.33500000000000002},"max":{"1min":178.41999999999999,"5min":293.48000000000002,"15min":293.48000000000002},"last":0.44500000000000001}]},{"osd":4,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":13.125,"5min":9.7409999999999997,"15min":18.032},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.22900000000000001},"max":{"1min":169.56100000000001,"5min":169.56100000000001,"15min":213.69999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":6.306,"5min":18.202999999999999,"15min":14.609999999999999},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":54.042999999999999,"5min":256.87799999999999,"15min":280.24599999999998},"last":0.33000000000000002}]},{"osd":5,"last update":"Wed Jan 25 11:16:27 2023","interfaces":[{"interface":"back","average":{"1min":15.02,"5min":23.286000000000001,"15min":19.399000000000001},"min":{"1min":0.308,"5min":0.28699999999999998,"15min":0.27500000000000002},"max":{"1min":86.936000000000007,"5min":231.13300000000001,"15min":231.13300000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.486,"5min":8.4870000000000001,"15min":10.06},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":22.335000000000001,"5min":145.10400000000001,"15min":162.10599999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Wed Jan 25 11:16:26 2023","interfaces":[{"interface":"back","average":{"1min":3.218,"5min":6.8760000000000003,"15min":8.1980000000000004},"min":{"1min":0.32500000000000001,"5min":0.28399999999999997,"15min":0.27800000000000002},"max":{"1min":58.448999999999998,"5min":153.49100000000001,"15min":153.49100000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.2269999999999999,"5min":9.7449999999999992,"15min":6.9320000000000004},"min":{"1min":0.30599999999999999,"5min":0.27100000000000002,"15min":0.27000000000000002},"max":{"1min":58.460999999999999,"5min":153.45699999999999,"15min":153.45699999999999},"last":0.42199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":25008,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94346832,"statfs":{"total":96636764160,"available":96611155968,"internally_reserved":0,"allocated":827392,"data_stored":354874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":27.518999999999998,"5min":21.922000000000001,"15min":21.248999999999999},"min":{"1min":0.36499999999999999,"5min":0.27300000000000002,"15min":0.21299999999999999},"max":{"1min":146.20099999999999,"5min":254.50299999999999,"15min":339.80599999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":26.100000000000001,"5min":14.125,"15min":13.346},"min":{"1min":0.35499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":281.52600000000001,"5min":281.52600000000001,"15min":281.52600000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":17.318000000000001,"5min":11.913,"15min":15.593},"min":{"1min":0.34399999999999997,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":183.36699999999999,"5min":183.36699999999999,"15min":256.67200000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":10.452999999999999,"5min":10.468,"15min":8.8740000000000006},"min":{"1min":0.378,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":85.468999999999994,"5min":114.57299999999999,"15min":195.858},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Jan 25 11:16:10 2023","interfaces":[{"interface":"back","average":{"1min":4.8019999999999996,"5min":6.048,"15min":6.9039999999999999},"min":{"1min":0.41099999999999998,"5min":0.34000000000000002,"15min":0.253},"max":{"1min":54.823,"5min":107.956,"15min":195.90899999999999},"last":0.626},{"interface":"front","average":{"1min":55.298000000000002,"5min":20.795000000000002,"15min":18.675999999999998},"min":{"1min":0.33100000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":377.28500000000003,"5min":377.28500000000003,"15min":377.28500000000003},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":58.204000000000001,"5min":34.033000000000001,"15min":22.539000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":316.32400000000001,"5min":329.22399999999999,"15min":329.22399999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":19.952999999999999,"5min":16.109999999999999,"15min":15.555},"min":{"1min":0.377,"5min":0.29599999999999999,"15min":0.26700000000000002},"max":{"1min":183.39599999999999,"5min":183.39599999999999,"15min":301.74200000000002},"last":0.63900000000000001}]},{"osd":4,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":16.503,"5min":8.0030000000000001,"15min":8.8759999999999994},"min":{"1min":0.30199999999999999,"5min":0.26000000000000001,"15min":0.222},"max":{"1min":165.761,"5min":165.761,"15min":195.93899999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":36.630000000000003,"5min":18.306000000000001,"15min":17.878},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.248},"max":{"1min":323.67200000000003,"5min":323.67200000000003,"15min":323.67200000000003},"last":0.50900000000000001}]},{"osd":5,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":15.914999999999999,"5min":9.298,"15min":14.316000000000001},"min":{"1min":0.27000000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":183.33699999999999,"5min":183.33699999999999,"15min":232.00700000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":7.8140000000000001,"5min":11.016,"15min":11.502000000000001},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.27900000000000003},"max":{"1min":54.789999999999999,"5min":184.25200000000001,"15min":291.60000000000002},"last":0.45900000000000002}]},{"osd":7,"last update":"Wed Jan 25 11:16:09 2023","interfaces":[{"interface":"back","average":{"1min":26.053000000000001,"5min":12.672000000000001,"15min":10.465999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28499999999999998},"max":{"1min":281.62,"5min":281.62,"15min":281.62},"last":0.442},{"interface":"front","average":{"1min":16.683,"5min":20.645,"15min":21.347999999999999},"min":{"1min":0.32000000000000001,"5min":0.29199999999999998,"15min":0.26100000000000001},"max":{"1min":183.29599999999999,"5min":183.29599999999999,"15min":329.608},"last":0.53500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29748,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":28928,"kb_avail":94342092,"statfs":{"total":96636764160,"available":96606302208,"internally_reserved":0,"allocated":831488,"data_stored":355155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29622272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":17.122,"5min":17.757000000000001,"15min":16.099},"min":{"1min":0.26200000000000001,"5min":0.217,"15min":0.19900000000000001},"max":{"1min":198.08199999999999,"5min":200.315,"15min":200.315},"last":0.46800000000000003},{"interface":"front","average":{"1min":4.3399999999999999,"5min":8.6910000000000007,"15min":14.369999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":50.058,"5min":175.48599999999999,"15min":261.98200000000003},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":7.4100000000000001,"15min":10.912000000000001},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24199999999999999},"max":{"1min":0.42199999999999999,"5min":181.09899999999999,"15min":181.09899999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":28.797999999999998,"5min":22.984000000000002,"15min":15.314},"min":{"1min":0.29599999999999999,"5min":0.24299999999999999,"15min":0.23000000000000001},"max":{"1min":95.108000000000004,"5min":239.709,"15min":239.709},"last":0.56899999999999995}]},{"osd":3,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":9.6959999999999997,"5min":19.744,"15min":16.367999999999999},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":97.210999999999999,"5min":182.119,"15min":266.98700000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":2.754,"5min":11.243,"15min":17.622},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":50.094999999999999,"5min":252.94399999999999,"15min":294.005},"last":0.59099999999999997}]},{"osd":4,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":3.3500000000000001,"5min":6.2649999999999997,"15min":13.388999999999999},"min":{"1min":0.37,"5min":0.32200000000000001,"15min":0.29899999999999999},"max":{"1min":59.338999999999999,"5min":100.40000000000001,"15min":259.99099999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":4.4020000000000001,"5min":18.411999999999999,"15min":12.458},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.35099999999999998},"max":{"1min":80.903999999999996,"5min":212.464,"15min":212.464},"last":0.68400000000000005}]},{"osd":5,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":10.648999999999999,"15min":16.501000000000001},"min":{"1min":0.39700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":50.256999999999998,"5min":146.64699999999999,"15min":262.11799999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":4.7089999999999996,"15min":13.087},"min":{"1min":0.42999999999999999,"5min":0.35899999999999999,"15min":0.34699999999999998},"max":{"1min":0.69999999999999996,"5min":100.36,"15min":177.58799999999999},"last":0.753}]},{"osd":6,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":12.359,"15min":7.0919999999999996},"min":{"1min":0.45200000000000001,"5min":0.39300000000000002,"15min":0.34100000000000003},"max":{"1min":0.751,"5min":181.80600000000001,"15min":181.80600000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":19.236000000000001,"5min":14.813000000000001,"15min":16.228000000000002},"min":{"1min":0.46999999999999997,"5min":0.38800000000000001,"15min":0.378},"max":{"1min":235.804,"5min":249.042,"15min":479.07799999999997},"last":0.76800000000000002}]},{"osd":7,"last update":"Wed Jan 25 11:16:33 2023","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":7.3630000000000004,"15min":13.981},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.37},"max":{"1min":50.210999999999999,"5min":146.613,"15min":275.94400000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":6.9089999999999998,"5min":21.227,"15min":19.664000000000001},"min":{"1min":0.46100000000000002,"5min":0.435,"15min":0.38700000000000001},"max":{"1min":86.629000000000005,"5min":346.58199999999999,"15min":346.58199999999999},"last":0.69799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575039,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":4708,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":9433493