2023-09-30T00:51:52.321 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:51:52.321 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857 --verbose --description rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/orig.config.yaml 2023-09-30T00:51:52.353 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:51:52.429 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7406857' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYhXcNqehS9ibtgkT9dvrVPoGR12Jzkh5yUUUPDQnQYb91PYMztoOBzxgImY6KdFoj+KioZ6uczc6LRFMBJuGM= tasks: - install: null - ceph: log-ignorelist: - but it is still running - had wrong client addr - had wrong cluster addr - must scrub before tier agent can activate - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping updates - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SMALLER_PG_NUM\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(FS_WITH_FAILED_MDS\) - \(FS_DEGRADED\) - \(POOL_BACKFILLFULL\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(POOL_NEARFULL\) - \(POOL_APP_NOT_ENABLED\) - \(AUTH_BAD_CAPS\) - \(FS_INLINE_DATA_DEPRECATED\) - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - workunit: clients: all: - cephtool - mon/pool_ops.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:51:52.430 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:51:52.430 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:51:52.430 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:51:52.432 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:51:52.453 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:51:52.453 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:51:52.454 INFO:teuthology.packaging:ref: None 2023-09-30T00:51:52.454 INFO:teuthology.packaging:tag: None 2023-09-30T00:51:52.454 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:51:52.454 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:51:52.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:51:52.596 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:51:52.596 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:51:52.602 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:51:52.602 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:51:52.646 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:51:52.660 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:51:52.680 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:51:52.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:46:55.339175', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYhXcNqehS9ibtgkT9dvrVPoGR12Jzkh5yUUUPDQnQYb91PYMztoOBzxgImY6KdFoj+KioZ6uczc6LRFMBJuGM='} 2023-09-30T00:51:52.707 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:51:52.715 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-30T00:51:52.715 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:51:52.782 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f45c7793ee0>, signals=[15]) 2023-09-30T00:51:52.782 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:51:52.846 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:51:52.846 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T00:51:52.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:51:52.916 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:51:52.922 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2023-09-30T00:51:52.988 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2023-09-30T00:51:52.988 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2023-09-30T00:51:53.049 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="8" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="8" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:51:53.050 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:51:53.051 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:51:53.051 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:51:53.051 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2023-09-30T00:51:53.076 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:51:53.084 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:51:53.106 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:51:53.106 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:51:53.126 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:51:53.138 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:51:53.138 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:51:53.192 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:51:53.228 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:51:53.248 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:51:53.249 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:51:53.249 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:51:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:51:53.269 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2023-09-30T00:51:53.329 INFO:teuthology.orchestra.run.smithi079.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:51:53.330 INFO:teuthology.task.kernel:Running kernel on smithi079: 4.18.0-489.el8.x86_64 2023-09-30T00:51:53.330 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2023-09-30T00:51:57.270 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - AppStream 25 MB/s | 33 MB 00:01 2023-09-30T00:52:04.719 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - BaseOS 34 MB/s | 48 MB 00:01 2023-09-30T00:52:12.193 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - Extras 31 kB/s | 18 kB 00:00 2023-09-30T00:52:12.919 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-30T00:52:13.220 INFO:teuthology.orchestra.run.smithi079.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-09-30T00:52:14.904 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-09-30T00:52:18.720 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-09-30T00:52:22.194 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:52:22.194 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:52:22.453 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T00:52:22.453 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout:Install 3 Packages 2023-09-30T00:52:22.454 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T00:52:22.455 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 88 M 2023-09-30T00:52:22.455 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 96 M 2023-09-30T00:52:22.455 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T00:52:23.163 INFO:teuthology.orchestra.run.smithi079.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 19 MB/s | 10 MB 00:00 2023-09-30T00:52:23.914 INFO:teuthology.orchestra.run.smithi079.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 28 MB/s | 35 MB 00:01 2023-09-30T00:52:24.372 INFO:teuthology.orchestra.run.smithi079.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 25 MB/s | 43 MB 00:01 2023-09-30T00:52:24.373 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:52:24.373 INFO:teuthology.orchestra.run.smithi079.stdout:Total 46 MB/s | 88 MB 00:01 2023-09-30T00:52:25.044 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T00:52:25.465 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T00:52:25.465 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T00:52:26.688 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T00:52:26.708 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T00:52:29.445 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T00:52:29.924 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:52:33.467 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:52:33.856 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:52:37.243 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:52:37.579 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:52:59.416 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:52:59.438 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:52:59.781 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:52:59.781 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:52:59.781 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T00:53:00.550 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T00:53:00.674 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:53:00.885 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:53:00.885 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:53:00.886 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:53:00.886 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:53:00.886 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:53:00.886 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:53:00.886 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:53:00.886 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:53:01.094 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:53:01.095 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:53:01.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2023-09-30T00:53:01.128 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.02 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 129.el8 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Group : System Environment/Base 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 9265586 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:53:01.129 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Relocations : (not relocatable) 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : CentOS BuildSystem 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T00:53:01.130 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:53:01.131 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:53:01.131 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:53:01.131 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2023-09-30T00:53:01.131 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T00:53:01.131 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:53:01.132 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:53:01.132 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:53:01.132 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:53:02.794 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2023-09-30T00:53:04.705 INFO:teuthology.orchestra.run.smithi079.stderr:done 2023-09-30T00:53:04.707 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2023-09-30T00:53:04.729 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.T946idFwhG 2023-09-30T00:53:04.730 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.T946idFwhG 2023-09-30T00:53:04.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.T946idFwhG 2023-09-30T00:53:04.922 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.T946idFwhG is 6KB 2023-09-30T00:53:04.971 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.T946idFwhG 2023-09-30T00:53:04.988 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:53:05.060 INFO:teuthology.orchestra.run.smithi079.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:53:05.061 INFO:teuthology.orchestra.run.smithi079.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:53:05.061 INFO:teuthology.orchestra.run.smithi079.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:53:05.061 INFO:teuthology.orchestra.run.smithi079.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:53:05.062 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:53:05.231 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2023-09-30T00:53:35.263 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:53:35.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-09-30T00:53:35.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:53:53.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2023-09-30T00:54:02.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-09-30T00:54:02.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:54:05.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2023-09-30T00:54:17.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-09-30T00:54:17.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:54:49.955 DEBUG:teuthology.orchestra.run.smithi079:> true 2023-09-30T00:54:50.601 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-09-30T00:54:50.601 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:54:50.602 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2023-09-30T00:54:50.620 INFO:teuthology.orchestra.run.smithi079.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:54:50.621 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:54:50.621 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:54:50.621 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:54:51.622 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:54:51.623 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:54:51.726 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2023-09-30T00:54:51.738 DEBUG:teuthology.parallel:result is None 2023-09-30T00:54:51.738 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:54:51.746 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:54:51.746 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:54:51.767 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:54:51.795 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:54:51.845 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:54:51.846 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:54:51.890 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:54:51.898 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:54:51.898 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:54:51.968 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:54:51.980 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:54:51.981 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:54:51.990 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:54:51.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:54:52.054 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:54:52.063 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:54:52.063 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:54:52.115 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:54:52.202 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:54:52.260 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:54:52.306 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:54:52.350 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T00:54:52.350 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:54:52.419 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2023-09-30T00:54:52.519 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:54:52.536 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:54:52.544 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:54:52.544 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:54:52.602 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:54:52.647 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:54:52.647 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2023-09-30T00:54:52.722 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2023-09-30T00:54:52.723 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:54:52.724 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2023-09-30T00:54:52.800 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2023-09-30T00:54:52.800 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:54:52.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:54:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:52.828 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:54:52.829 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:54:52.896 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:54:52.905 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:54:52.905 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:54:52.922 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:54:52.923 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T01:02:21.341 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com')] 2023-09-30T01:02:21.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-09-30T01:02:21.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:02:21.411 DEBUG:teuthology.orchestra.run.smithi079:> true 2023-09-30T01:02:21.484 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2023-09-30T01:02:21.485 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T01:02:21.495 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T01:02:21.496 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:02:21.496 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:02:21.557 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T01:02:21.572 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T01:02:21.639 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2023-09-30T01:02:21.650 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2023-09-30T01:02:21.664 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T01:02:21.677 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T01:02:21.745 INFO:teuthology.orchestra.run.smithi079.stderr:bash: ntpq: command not found 2023-09-30T01:02:21.750 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:02:21.750 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2023-09-30T01:02:21.750 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:02:21.750 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:02:21.750 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:02:21.750 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:02:21.751 INFO:teuthology.run_tasks:Running task install... 2023-09-30T01:02:21.760 DEBUG:teuthology.task.install:project ceph 2023-09-30T01:02:21.761 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T01:02:21.761 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T01:02:21.761 INFO:teuthology.task.install:Using flavor: default 2023-09-30T01:02:21.767 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T01:02:21.767 INFO:teuthology.task.install:extra packages: [] 2023-09-30T01:02:21.767 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T01:02:21.768 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:02:21.986 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T01:02:21.986 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T01:02:22.094 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T01:02:22.095 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:02:22.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T01:02:22.124 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T01:02:22.124 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T01:02:22.193 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T01:02:22.271 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T01:02:22.338 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2023-09-30T01:02:22.339 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2023-09-30T01:02:22.608 INFO:teuthology.orchestra.run.smithi079.stdout:76 files removed 2023-09-30T01:02:22.636 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2023-09-30T01:02:23.399 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 217 kB/s | 78 kB 00:00 2023-09-30T01:02:23.808 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 25 kB/s | 9.2 kB 00:00 2023-09-30T01:02:24.459 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2023-09-30T01:02:26.201 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - AppStream 19 MB/s | 33 MB 00:01 2023-09-30T01:02:33.859 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - BaseOS 32 MB/s | 48 MB 00:01 2023-09-30T01:02:41.077 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2023-09-30T01:02:41.668 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T01:02:43.730 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 8 - PowerTools 3.4 MB/s | 6.5 MB 00:01 2023-09-30T01:02:45.257 INFO:teuthology.orchestra.run.smithi079.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2023-09-30T01:02:49.560 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 16 MB 00:04 2023-09-30T01:02:53.421 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 257 kB/s | 24 kB 00:00 2023-09-30T01:02:57.808 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:02:57.810 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:02:57.810 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T01:02:57.811 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T01:02:57.812 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout:Install 21 Packages 2023-09-30T01:02:57.813 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2023-09-30T01:02:57.814 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:02:57.814 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 63 M 2023-09-30T01:02:57.814 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:03:00.217 INFO:teuthology.orchestra.run.smithi079.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 2.8 MB/s | 5.2 MB 00:01 2023-09-30T01:03:00.311 INFO:teuthology.orchestra.run.smithi079.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 266 kB/s | 24 kB 00:00 2023-09-30T01:03:00.500 INFO:teuthology.orchestra.run.smithi079.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 4.1 MB/s | 796 kB 00:00 2023-09-30T01:03:00.634 INFO:teuthology.orchestra.run.smithi079.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 3.7 MB/s | 506 kB 00:00 2023-09-30T01:03:01.277 INFO:teuthology.orchestra.run.smithi079.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 8.7 MB/s | 5.6 MB 00:00 2023-09-30T01:03:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:(6/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 5.4 MB/s | 17 MB 00:03 2023-09-30T01:03:01.544 INFO:teuthology.orchestra.run.smithi079.stdout:(7/23): python3-ceph-argparse-17.2.6-1165.g93dc 167 kB/s | 45 kB 00:00 2023-09-30T01:03:01.594 INFO:teuthology.orchestra.run.smithi079.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 1.5 MB/s | 115 kB 00:00 2023-09-30T01:03:01.641 INFO:teuthology.orchestra.run.smithi079.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.9 MB/s | 190 kB 00:00 2023-09-30T01:03:01.678 INFO:teuthology.orchestra.run.smithi079.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 4.3 MB/s | 370 kB 00:00 2023-09-30T01:03:01.728 INFO:teuthology.orchestra.run.smithi079.stdout:(11/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 4.0 MB/s | 358 kB 00:00 2023-09-30T01:03:01.754 INFO:teuthology.orchestra.run.smithi079.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.4 MB/s | 107 kB 00:00 2023-09-30T01:03:02.020 INFO:teuthology.orchestra.run.smithi079.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.2 MB/s | 354 kB 00:00 2023-09-30T01:03:02.046 INFO:teuthology.orchestra.run.smithi079.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 889 kB/s | 259 kB 00:00 2023-09-30T01:03:02.080 INFO:teuthology.orchestra.run.smithi079.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 749 kB/s | 44 kB 00:00 2023-09-30T01:03:02.213 INFO:teuthology.orchestra.run.smithi079.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 281 kB/s | 47 kB 00:00 2023-09-30T01:03:02.764 INFO:teuthology.orchestra.run.smithi079.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 107 kB/s | 59 kB 00:00 2023-09-30T01:03:02.856 INFO:teuthology.orchestra.run.smithi079.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 395 kB/s | 306 kB 00:00 2023-09-30T01:03:02.964 INFO:teuthology.orchestra.run.smithi079.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 375 kB/s | 75 kB 00:00 2023-09-30T01:03:03.365 INFO:teuthology.orchestra.run.smithi079.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 3.4 MB/s | 1.7 MB 00:00 2023-09-30T01:03:03.465 INFO:teuthology.orchestra.run.smithi079.stdout:(21/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 6.9 MB/s | 3.5 MB 00:00 2023-09-30T01:03:03.840 INFO:teuthology.orchestra.run.smithi079.stdout:(22/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 4.3 MB/s | 23 MB 00:05 2023-09-30T01:03:03.974 INFO:teuthology.orchestra.run.smithi079.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 5.9 MB/s | 3.6 MB 00:00 2023-09-30T01:03:03.977 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:03:03.978 INFO:teuthology.orchestra.run.smithi079.stdout:Total 10 MB/s | 63 MB 00:06 2023-09-30T01:03:04.019 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:03:04.064 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:03:04.065 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:03:04.440 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:03:04.443 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:03:05.634 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:03:05.869 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T01:03:06.073 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T01:03:06.455 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T01:03:06.674 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:03:06.765 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:03:07.048 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:03:07.240 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:03:07.672 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:03:07.981 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:03:08.197 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:03:08.394 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T01:03:08.587 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T01:03:08.773 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T01:03:09.313 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T01:03:09.481 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:03:09.544 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:03:09.733 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T01:03:09.918 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:03:10.158 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:03:10.393 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T01:03:10.572 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:03:10.642 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:03:10.890 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T01:03:11.106 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T01:03:11.273 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T01:03:11.473 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T01:03:14.180 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:03:14.414 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:03:14.996 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:03:15.279 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:03:15.413 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:03:15.563 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:03:34.218 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:03:34.218 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2023-09-30T01:03:34.218 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /dev 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /run 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /dev 2023-09-30T01:03:34.219 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:35.941 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:03:36.018 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:03:36.018 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:03:36.269 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:03:36.269 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:03:37.032 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T01:03:37.033 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T01:03:37.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T01:03:37.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:03:37.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:03:37.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:03:38.194 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:03:38.195 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:38.195 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2023-09-30T01:03:38.195 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.196 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:03:38.197 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:38.198 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:03:38.532 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2023-09-30T01:03:39.387 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:46 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:03:40.970 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:03:40.971 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T01:03:40.972 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T01:03:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:40.973 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:03:40.973 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:03:40.973 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2023-09-30T01:03:40.973 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:40.973 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 53 M 2023-09-30T01:03:40.974 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 233 M 2023-09-30T01:03:40.974 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:03:41.394 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 902 kB/s | 202 kB 00:00 2023-09-30T01:03:41.503 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-09-30T01:03:41.528 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 488 kB/s | 175 kB 00:00 2023-09-30T01:03:41.654 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2023-09-30T01:03:41.712 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 384 kB/s | 71 kB 00:00 2023-09-30T01:03:43.696 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 52 MB 00:02 2023-09-30T01:03:43.698 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:03:43.698 INFO:teuthology.orchestra.run.smithi079.stdout:Total 19 MB/s | 53 MB 00:02 2023-09-30T01:03:43.741 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:03:43.763 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:03:43.763 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:03:44.075 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:03:44.078 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:03:44.581 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:03:44.855 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T01:03:45.086 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T01:03:45.265 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:03:45.342 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:03:45.564 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T01:03:45.723 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:03:50.737 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:03:50.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:03:51.277 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:03:51.277 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:03:51.277 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:03:51.277 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T01:03:51.277 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T01:03:51.277 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T01:03:51.935 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T01:03:51.935 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:51.935 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:03:51.935 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:51.935 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-7.el8.x86_64 2023-09-30T01:03:51.936 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:03:51.936 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:03:51.936 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T01:03:51.936 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:03:51.936 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:51.936 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:03:52.142 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2023-09-30T01:03:52.998 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:03:54.576 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:03:54.579 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:03:54.579 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:03:54.579 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:03:54.579 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:03:54.580 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:03:54.581 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T01:03:54.582 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:03:54.583 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:03:54.584 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T01:03:54.584 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:03:54.584 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:54.585 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:03:54.585 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:03:54.585 INFO:teuthology.orchestra.run.smithi079.stdout:Install 32 Packages 2023-09-30T01:03:54.585 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:03:54.586 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 28 M 2023-09-30T01:03:54.586 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 110 M 2023-09-30T01:03:54.586 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:03:54.907 INFO:teuthology.orchestra.run.smithi079.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 32 kB/s | 6.4 kB 00:00 2023-09-30T01:03:55.199 INFO:teuthology.orchestra.run.smithi079.stdout:(2/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 4.7 MB/s | 2.3 MB 00:00 2023-09-30T01:03:55.499 INFO:teuthology.orchestra.run.smithi079.stdout:(3/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 2.0 MB/s | 1.6 MB 00:00 2023-09-30T01:03:55.616 INFO:teuthology.orchestra.run.smithi079.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.0 MB/s | 238 kB 00:00 2023-09-30T01:03:55.933 INFO:teuthology.orchestra.run.smithi079.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 495 kB/s | 157 kB 00:00 2023-09-30T01:03:55.992 INFO:teuthology.orchestra.run.smithi079.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 667 kB/s | 39 kB 00:00 2023-09-30T01:03:56.092 INFO:teuthology.orchestra.run.smithi079.stdout:(7/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 3.3 MB/s | 3.9 MB 00:01 2023-09-30T01:03:56.117 INFO:teuthology.orchestra.run.smithi079.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 826 kB/s | 103 kB 00:00 2023-09-30T01:03:56.335 INFO:teuthology.orchestra.run.smithi079.stdout:(9/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 15 MB/s | 17 MB 00:01 2023-09-30T01:03:56.469 INFO:teuthology.orchestra.run.smithi079.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 152 kB/s | 57 kB 00:00 2023-09-30T01:03:56.494 INFO:teuthology.orchestra.run.smithi079.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2023-09-30T01:03:56.586 INFO:teuthology.orchestra.run.smithi079.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 993 kB/s | 249 kB 00:00 2023-09-30T01:03:56.686 INFO:teuthology.orchestra.run.smithi079.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 567 kB/s | 109 kB 00:00 2023-09-30T01:03:56.778 INFO:teuthology.orchestra.run.smithi079.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.0 MB/s | 641 kB 00:00 2023-09-30T01:03:56.835 INFO:teuthology.orchestra.run.smithi079.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 175 kB/s | 44 kB 00:00 2023-09-30T01:03:57.004 INFO:teuthology.orchestra.run.smithi079.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 585 kB/s | 185 kB 00:00 2023-09-30T01:03:57.046 INFO:teuthology.orchestra.run.smithi079.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.8 MB/s | 384 kB 00:00 2023-09-30T01:03:57.105 INFO:teuthology.orchestra.run.smithi079.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2023-09-30T01:03:57.138 INFO:teuthology.orchestra.run.smithi079.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2023-09-30T01:03:57.163 INFO:teuthology.orchestra.run.smithi079.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 449 kB/s | 173 kB 00:00 2023-09-30T01:03:57.189 INFO:teuthology.orchestra.run.smithi079.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 585 kB/s | 49 kB 00:00 2023-09-30T01:03:57.214 INFO:teuthology.orchestra.run.smithi079.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2023-09-30T01:03:57.248 INFO:teuthology.orchestra.run.smithi079.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-09-30T01:03:57.273 INFO:teuthology.orchestra.run.smithi079.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 319 kB/s | 19 kB 00:00 2023-09-30T01:03:57.307 INFO:teuthology.orchestra.run.smithi079.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2023-09-30T01:03:57.332 INFO:teuthology.orchestra.run.smithi079.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 500 kB/s | 29 kB 00:00 2023-09-30T01:03:57.391 INFO:teuthology.orchestra.run.smithi079.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-09-30T01:03:57.416 INFO:teuthology.orchestra.run.smithi079.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2023-09-30T01:03:57.466 INFO:teuthology.orchestra.run.smithi079.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2023-09-30T01:03:57.491 INFO:teuthology.orchestra.run.smithi079.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-09-30T01:03:57.525 INFO:teuthology.orchestra.run.smithi079.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 393 kB/s | 23 kB 00:00 2023-09-30T01:03:57.583 INFO:teuthology.orchestra.run.smithi079.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-09-30T01:03:57.587 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:03:57.588 INFO:teuthology.orchestra.run.smithi079.stdout:Total 9.5 MB/s | 28 MB 00:03 2023-09-30T01:03:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:03:57.709 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:03:57.709 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:03:58.016 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:03:58.019 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:03:59.243 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:03:59.442 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T01:03:59.686 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T01:03:59.920 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T01:04:00.151 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T01:04:00.366 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T01:04:00.605 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T01:04:00.773 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T01:04:01.009 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T01:04:01.227 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T01:04:01.454 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T01:04:01.717 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T01:04:01.934 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T01:04:02.307 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T01:04:02.597 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T01:04:02.830 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T01:04:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T01:04:03.312 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:04:03.637 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T01:04:03.866 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T01:04:04.104 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T01:04:04.353 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T01:04:04.591 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T01:04:04.831 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T01:04:05.074 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T01:04:05.401 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T01:04:05.716 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T01:04:06.098 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T01:04:06.351 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:04:07.837 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:04:08.001 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:04:08.393 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:04:08.593 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:04:08.810 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:04:08.985 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:04:09.003 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:04:09.151 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:04:09.546 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T01:04:09.547 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T01:04:09.548 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T01:04:09.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T01:04:09.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T01:04:09.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T01:04:09.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T01:04:09.549 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T01:04:09.550 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T01:04:09.550 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T01:04:09.550 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T01:04:09.550 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T01:04:11.235 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:04:11.236 INFO:teuthology.orchestra.run.smithi079.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:04:11.237 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:04:11.238 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:04:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:04:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:04:11.239 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:11.239 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:11.507 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2023-09-30T01:04:12.355 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:19 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:13.746 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:04:13.932 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:13.934 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:04:13.934 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:14.027 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2023-09-30T01:04:14.867 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:21 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:16.434 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:16.435 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:16.435 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:04:16.435 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:16.435 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:04:16.435 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:04:16.436 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:16.436 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:04:16.436 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:16.436 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:04:16.436 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:16.437 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 80 k 2023-09-30T01:04:16.437 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 363 k 2023-09-30T01:04:16.437 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:04:16.678 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 331 kB/s | 80 kB 00:00 2023-09-30T01:04:16.679 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:04:16.679 INFO:teuthology.orchestra.run.smithi079.stdout:Total 329 kB/s | 80 kB 00:00 2023-09-30T01:04:16.679 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:04:16.687 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:04:16.687 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:04:16.689 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:04:16.692 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:04:16.760 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:04:17.294 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:04:17.445 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:04:17.663 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:04:18.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:04:18.283 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:18.283 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:04:18.283 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:18.283 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:18.283 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:18.536 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:04:19.393 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:26 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:20.972 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:20.973 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:20.973 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2023-09-30T01:04:20.973 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:20.973 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:04:20.973 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 161 k 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 490 k 2023-09-30T01:04:20.974 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:04:21.543 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 284 kB/s | 161 kB 00:00 2023-09-30T01:04:21.543 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:04:21.544 INFO:teuthology.orchestra.run.smithi079.stdout:Total 283 kB/s | 161 kB 00:00 2023-09-30T01:04:21.544 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:04:21.555 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:04:21.555 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:04:21.605 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:04:21.607 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:04:22.245 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:04:23.073 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:04:23.533 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:04:24.124 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:04:24.125 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:24.125 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:04:24.125 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:04:24.125 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:24.125 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:24.359 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2023-09-30T01:04:25.204 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:26.584 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:04:26.771 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:26.772 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:04:26.772 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:26.867 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:04:27.711 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:29.283 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:29.284 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:29.284 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:04:29.284 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:29.284 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:04:29.284 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2023-09-30T01:04:29.285 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:29.286 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.0 M 2023-09-30T01:04:29.286 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 8.0 M 2023-09-30T01:04:29.286 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:04:29.622 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 96 kB/s | 14 kB 00:00 2023-09-30T01:04:29.723 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 92 kB/s | 23 kB 00:00 2023-09-30T01:04:29.782 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.5 MB/s | 1.7 MB 00:00 2023-09-30T01:04:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2023-09-30T01:04:29.899 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 191 kB/s | 34 kB 00:00 2023-09-30T01:04:30.058 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 713 kB/s | 196 kB 00:00 2023-09-30T01:04:30.059 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:04:30.059 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-09-30T01:04:30.069 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:04:30.078 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:04:30.078 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:04:30.163 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:04:30.165 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:04:30.445 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:04:30.669 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T01:04:30.872 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T01:04:31.057 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T01:04:31.211 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:04:31.523 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:04:31.754 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:04:32.038 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:04:32.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:04:32.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:04:32.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:04:32.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T01:04:32.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T01:04:32.660 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:32.661 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:32.820 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:04:33.668 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:35.252 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:35.253 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================================ 2023-09-30T01:04:35.253 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:04:35.253 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================================ 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T01:04:35.254 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================================ 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout:Install 8 Packages 2023-09-30T01:04:35.255 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:35.256 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 35 M 2023-09-30T01:04:35.256 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 210 M 2023-09-30T01:04:35.256 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:04:36.315 INFO:teuthology.orchestra.run.smithi079.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-116 7.5 MB/s | 7.4 MB 00:00 2023-09-30T01:04:36.728 INFO:teuthology.orchestra.run.smithi079.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 9.0 MB/s | 3.7 MB 00:00 2023-09-30T01:04:36.790 INFO:teuthology.orchestra.run.smithi079.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.5 MB/s | 225 kB 00:00 2023-09-30T01:04:36.874 INFO:teuthology.orchestra.run.smithi079.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.2 MB/s | 4.9 MB 00:01 2023-09-30T01:04:36.933 INFO:teuthology.orchestra.run.smithi079.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 3.0 MB/s | 4.7 MB 00:01 2023-09-30T01:04:37.166 INFO:teuthology.orchestra.run.smithi079.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 738 kB/s | 172 kB 00:00 2023-09-30T01:04:37.333 INFO:teuthology.orchestra.run.smithi079.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 25 MB/s | 14 MB 00:00 2023-09-30T01:04:37.375 INFO:teuthology.orchestra.run.smithi079.stdout:(8/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2023-09-30T01:04:37.375 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:04:37.376 INFO:teuthology.orchestra.run.smithi079.stdout:Total 17 MB/s | 35 MB 00:02 2023-09-30T01:04:37.609 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:04:37.627 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:04:37.628 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:04:38.001 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:04:38.004 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:04:38.606 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:04:39.761 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:04:39.896 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:04:40.078 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:04:40.678 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:04:40.911 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:04:41.537 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:04:41.710 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:04:42.418 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:04:42.773 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:04:43.044 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:04:44.795 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:04:46.104 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T01:04:46.318 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T01:04:46.628 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T01:04:47.328 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T01:04:47.328 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:47.328 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:47.329 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:47.461 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:04:48.312 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:55 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:04:49.915 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:04:49.917 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:04:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:04:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T01:04:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T01:04:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:04:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T01:04:49.918 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:04:49.919 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2023-09-30T01:04:49.920 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:49.920 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 1.9 M 2023-09-30T01:04:49.920 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 25 M 2023-09-30T01:04:49.920 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:04:50.460 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 171 kB/s | 27 kB 00:00 2023-09-30T01:04:50.487 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 109 kB/s | 20 kB 00:00 2023-09-30T01:04:50.629 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 893 kB/s | 126 kB 00:00 2023-09-30T01:04:50.654 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 135 kB/s | 47 kB 00:00 2023-09-30T01:04:50.679 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 573 kB/s | 110 kB 00:00 2023-09-30T01:04:50.879 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 213 kB/s | 43 kB 00:00 2023-09-30T01:04:50.930 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 515 kB/s | 155 kB 00:00 2023-09-30T01:04:51.155 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 69 kB/s | 33 kB 00:00 2023-09-30T01:04:51.347 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 35 kB/s | 16 kB 00:00 2023-09-30T01:04:51.448 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 157 kB/s | 81 kB 00:00 2023-09-30T01:04:51.548 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 321 kB/s | 64 kB 00:00 2023-09-30T01:04:51.575 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 489 kB/s | 61 kB 00:00 2023-09-30T01:04:52.748 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 750 kB/s | 1.2 MB 00:01 2023-09-30T01:04:52.752 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:04:52.752 INFO:teuthology.orchestra.run.smithi079.stdout:Total 698 kB/s | 1.9 MB 00:02 2023-09-30T01:04:52.811 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:04:52.827 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:04:52.827 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:04:52.914 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:04:52.917 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:04:53.435 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:04:53.736 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:04:53.958 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T01:04:54.189 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T01:04:54.415 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T01:04:54.630 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T01:04:54.886 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T01:04:55.497 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T01:04:55.779 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T01:04:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T01:04:57.139 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T01:04:57.331 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T01:04:57.516 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T01:04:57.732 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T01:04:58.061 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T01:04:58.062 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T01:04:58.062 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:04:58.062 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T01:04:58.062 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T01:04:58.062 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T01:04:58.738 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:04:58.739 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:04:58.740 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:04:58.740 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:04:58.740 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:04:58.740 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:04:58.740 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:04:58.740 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:04:58.870 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:04:59.723 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:06 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:01.295 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:05:01.296 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout:Install 5 Packages 2023-09-30T01:05:01.297 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:01.298 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.0 M 2023-09-30T01:05:01.298 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2023-09-30T01:05:01.298 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:01.632 INFO:teuthology.orchestra.run.smithi079.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2023-09-30T01:05:01.657 INFO:teuthology.orchestra.run.smithi079.stdout:(2/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 420 kB/s | 121 kB 00:00 2023-09-30T01:05:01.833 INFO:teuthology.orchestra.run.smithi079.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2023-09-30T01:05:01.933 INFO:teuthology.orchestra.run.smithi079.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 219 kB/s | 60 kB 00:00 2023-09-30T01:05:02.075 INFO:teuthology.orchestra.run.smithi079.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2023-09-30T01:05:02.077 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:02.105 INFO:teuthology.orchestra.run.smithi079.stdout:Total 7.6 MB/s | 6.0 MB 00:00 2023-09-30T01:05:02.128 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:02.140 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:02.140 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:02.244 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:02.247 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:02.586 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:02.895 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T01:05:03.877 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:05:04.252 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T01:05:04.549 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T01:05:04.764 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:05:05.033 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:05:05.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:05:05.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T01:05:05.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T01:05:05.034 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:05:05.673 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T01:05:05.673 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:05.673 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:05.673 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:05:05.673 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:05:05.674 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T01:05:05.674 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T01:05:05.674 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:05:05.674 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:05.674 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:05.879 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2023-09-30T01:05:06.751 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:13 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:08.321 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:08.321 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:08.321 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2023-09-30T01:05:08.322 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:08.323 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 957 k 2023-09-30T01:05:08.323 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 3.0 M 2023-09-30T01:05:08.323 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:09.942 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 874 kB 00:00 2023-09-30T01:05:10.750 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 68 kB/s | 83 kB 00:01 2023-09-30T01:05:10.751 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:10.751 INFO:teuthology.orchestra.run.smithi079.stdout:Total 394 kB/s | 957 kB 00:02 2023-09-30T01:05:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:10.785 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:10.786 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:10.877 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:10.880 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:11.554 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:11.858 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T01:05:12.079 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:05:12.467 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:05:12.467 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:05:13.095 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T01:05:13.096 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:13.096 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:13.096 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T01:05:13.096 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:13.096 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:13.305 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2023-09-30T01:05:14.165 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:21 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:15.743 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:15.743 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:15.743 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2023-09-30T01:05:15.743 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:15.743 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:15.744 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:05:15.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:15.744 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:15.744 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:15.744 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:05:15.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:15.745 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 267 k 2023-09-30T01:05:15.745 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.3 M 2023-09-30T01:05:15.745 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:15.957 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.2 MB/s | 267 kB 00:00 2023-09-30T01:05:15.957 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:15.958 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.2 MB/s | 267 kB 00:00 2023-09-30T01:05:15.958 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:15.968 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:15.968 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:15.999 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:16.002 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:16.217 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:16.950 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:05:17.693 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:05:18.621 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:05:18.621 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:18.621 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:18.621 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:05:18.622 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:18.622 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:18.786 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2023-09-30T01:05:19.657 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:26 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:21.251 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:21.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:05:21.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:21.253 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 130 k 2023-09-30T01:05:21.253 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 464 k 2023-09-30T01:05:21.253 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:21.448 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 666 kB/s | 130 kB 00:00 2023-09-30T01:05:21.449 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:21.449 INFO:teuthology.orchestra.run.smithi079.stdout:Total 661 kB/s | 130 kB 00:00 2023-09-30T01:05:21.449 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:21.454 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:21.454 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:21.503 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:21.506 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:21.612 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:21.876 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:22.215 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:22.845 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:22.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:22.845 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:22.845 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:05:22.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:22.846 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:23.021 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2023-09-30T01:05:23.877 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:30 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:25.275 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:25.460 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:25.461 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:25.461 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:25.558 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2023-09-30T01:05:26.410 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:33 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:27.996 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:27.996 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2023-09-30T01:05:27.996 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:27.996 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:27.996 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 25 k 2023-09-30T01:05:27.997 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 109 k 2023-09-30T01:05:27.998 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:28.251 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 98 kB/s | 25 kB 00:00 2023-09-30T01:05:28.252 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:28.252 INFO:teuthology.orchestra.run.smithi079.stdout:Total 97 kB/s | 25 kB 00:00 2023-09-30T01:05:28.253 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:28.256 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:28.256 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:28.258 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:28.261 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:28.311 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:28.524 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:28.772 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:29.771 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:29.771 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:29.771 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:29.771 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:05:29.772 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:29.772 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:29.886 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2023-09-30T01:05:30.734 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:37 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:32.123 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:32.309 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:32.310 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:32.310 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:32.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2023-09-30T01:05:33.272 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:40 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:34.684 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:34.874 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:34.875 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:34.875 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:34.972 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2023-09-30T01:05:35.828 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:42 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:37.214 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:37.400 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:37.401 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:37.402 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:37.508 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2023-09-30T01:05:38.366 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:45 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:39.760 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:39.946 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:39.949 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:39.949 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:40.044 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2023-09-30T01:05:40.900 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:42.297 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:42.485 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:42.486 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:42.486 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:42.589 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2023-09-30T01:05:43.438 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:50 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:44.830 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:05:45.016 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:45.017 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2023-09-30T01:05:45.017 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:45.143 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2023-09-30T01:05:45.994 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:52 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:47.576 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:47.576 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:47.576 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:05:47.577 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:47.578 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 90 k 2023-09-30T01:05:47.578 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 264 k 2023-09-30T01:05:47.578 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:47.748 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 530 kB/s | 90 kB 00:00 2023-09-30T01:05:47.749 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:47.749 INFO:teuthology.orchestra.run.smithi079.stdout:Total 527 kB/s | 90 kB 00:00 2023-09-30T01:05:47.749 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:47.758 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:47.759 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:47.807 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:47.810 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:47.910 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:48.120 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:48.390 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:48.998 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:48.998 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:48.998 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:48.999 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:05:48.999 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:48.999 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:49.207 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2023-09-30T01:05:50.069 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:02:57 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:51.649 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:51.649 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:51.649 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:05:51.650 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:51.651 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.5 M 2023-09-30T01:05:51.651 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 16 M 2023-09-30T01:05:51.651 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:53.135 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 2.3 MB/s | 3.5 MB 00:01 2023-09-30T01:05:53.135 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:53.136 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.3 MB/s | 3.5 MB 00:01 2023-09-30T01:05:53.136 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:53.148 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:53.148 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:53.211 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:53.214 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:53.673 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:53.895 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:54.319 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:55.004 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:55.004 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:55.005 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:55.005 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:05:55.005 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:55.005 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:55.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2023-09-30T01:05:56.108 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:03:03 ago on Sat 30 Sep 2023 01:02:53 AM UTC. 2023-09-30T01:05:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2023-09-30T01:05:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:57.674 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2023-09-30T01:05:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2023-09-30T01:05:57.674 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:05:57.675 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2023-09-30T01:05:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2023-09-30T01:05:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2023-09-30T01:05:57.675 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 173 k 2023-09-30T01:05:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 552 k 2023-09-30T01:05:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2023-09-30T01:05:57.871 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 886 kB/s | 173 kB 00:00 2023-09-30T01:05:57.872 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:05:57.872 INFO:teuthology.orchestra.run.smithi079.stdout:Total 880 kB/s | 173 kB 00:00 2023-09-30T01:05:57.872 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2023-09-30T01:05:57.883 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2023-09-30T01:05:57.883 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2023-09-30T01:05:57.933 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2023-09-30T01:05:57.935 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2023-09-30T01:05:58.081 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2023-09-30T01:05:58.268 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:59.271 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:05:59.272 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:59.272 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2023-09-30T01:05:59.272 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:05:59.272 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:05:59.272 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2023-09-30T01:05:59.508 DEBUG:teuthology.parallel:result is None 2023-09-30T01:05:59.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:05:59.658 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T01:05:59.681 INFO:teuthology.orchestra.run.smithi079.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T01:05:59.681 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T01:05:59.681 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T01:05:59.682 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T01:05:59.682 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:05:59.682 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:05:59.749 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T01:05:59.749 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:05:59.749 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:05:59.816 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:05:59.884 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T01:05:59.885 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:05:59.885 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:05:59.952 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:06:00.019 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T01:06:00.019 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:00.019 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:06:00.086 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:06:00.153 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T01:06:00.217 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T01:06:00.218 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2023-09-30T01:06:00.245 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T01:06:00.245 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:06:00.312 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T01:06:00.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:06:00.388 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T01:06:00.388 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T01:06:00.389 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406857', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYhXcNqehS9ibtgkT9dvrVPoGR12Jzkh5yUUUPDQnQYb91PYMztoOBzxgImY6KdFoj+KioZ6uczc6LRFMBJuGM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['but it is still running', 'had wrong client addr', 'had wrong cluster addr', 'must scrub before tier agent can activate', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping updates', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SMALLER_PG_NUM\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(FS_DEGRADED\\)', '\\(POOL_BACKFILLFULL\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(POOL_NEARFULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(AUTH_BAD_CAPS\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['cephtool', 'mon/pool_ops.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T01:06:00.389 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:06:00.437 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:06:00.504 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:00.504 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:06:00.562 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:06:00.563 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 386386 Links: 1 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-09-30 01:05:54.073133042 +0000 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-09-30 01:01:41.923651543 +0000 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-09-30 01:01:41.923651543 +0000 2023-09-30T01:06:00.621 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-09-30T01:06:00.622 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:06:00.687 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-09-30T01:06:00.687 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-09-30T01:06:00.687 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000238256 s, 2.1 MB/s 2023-09-30T01:06:00.688 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:06:00.747 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 386432 Links: 1 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-09-30 01:05:54.073133042 +0000 2023-09-30T01:06:00.805 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-09-30 01:01:42.436640311 +0000 2023-09-30T01:06:00.806 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-09-30 01:01:42.436640311 +0000 2023-09-30T01:06:00.806 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-09-30T01:06:00.806 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:06:00.871 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-09-30T01:06:00.871 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-09-30T01:06:00.871 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000196067 s, 2.6 MB/s 2023-09-30T01:06:00.872 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:06:00.930 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 393896 Links: 1 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-09-30 01:05:54.073133042 +0000 2023-09-30T01:06:00.989 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-09-30 01:01:42.962628795 +0000 2023-09-30T01:06:00.990 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-09-30 01:01:42.962628795 +0000 2023-09-30T01:06:00.990 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-09-30T01:06:00.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:06:01.055 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-09-30T01:06:01.055 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-09-30T01:06:01.055 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000194153 s, 2.6 MB/s 2023-09-30T01:06:01.056 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:06:01.115 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2023-09-30T01:06:01.172 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 6h/6d Inode: 394689 Links: 1 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2023-09-30 01:05:54.073133042 +0000 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2023-09-30 01:01:43.484617366 +0000 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2023-09-30 01:01:43.484617366 +0000 2023-09-30T01:06:01.173 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: - 2023-09-30T01:06:01.173 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:06:01.239 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2023-09-30T01:06:01.239 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2023-09-30T01:06:01.239 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000220011 s, 2.3 MB/s 2023-09-30T01:06:01.240 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:06:01.299 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:06:01.299 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-30T01:06:01.299 INFO:tasks.ceph:Generating config... 2023-09-30T01:06:01.300 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T01:06:01.300 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T01:06:01.301 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T01:06:01.301 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T01:06:01.301 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T01:06:01.301 INFO:tasks.ceph:[global] ms type = async 2023-09-30T01:06:01.301 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T01:06:01.301 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T01:06:01.301 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T01:06:01.301 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T01:06:01.301 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T01:06:01.301 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T01:06:01.301 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T01:06:01.302 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T01:06:01.303 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T01:06:01.304 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T01:06:01.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T01:06:01.395 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T01:06:01.398 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T01:06:01.439 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:06:01.506 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.79'), ('mon.b', '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]'), ('mon.c', '[v2:172.21.15.79:3302,v1:172.21.15.79:6791]')] 2023-09-30T01:06:01.507 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.79,[v2:172.21.15.79:3301,v1:172.21.15.79:6790],[v2:172.21.15.79:3302,v1:172.21.15.79:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-09-30T01:06:01.507 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:01.507 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:06:01.564 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.79 --addv b '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]' --addv c '[v2:172.21.15.79:3302,v1:172.21.15.79:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:01.649 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: generated fsid 06bf7746-ed30-4322-b1bc-267eae5b6611 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = octopus 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 0 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:fsid 06bf7746-ed30-4322-b1bc-267eae5b6611 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:last_changed 2023-09-30T01:06:01.650264+0000 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:created 2023-09-30T01:06:01.650264+0000 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:min_mon_release 15 (octopus) 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:election_strategy: 3 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:0: [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon.a 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:1: [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon.b 2023-09-30T01:06:01.650 INFO:teuthology.orchestra.run.smithi079.stdout:2: [v2:172.21.15.79:3302/0,v1:172.21.15.79:6791/0] mon.c 2023-09-30T01:06:01.651 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T01:06:01.652 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:06:01.669 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 06bf7746-ed30-4322-b1bc-267eae5b6611... 2023-09-30T01:06:01.670 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2023-09-30T01:06:01.754 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:06:01.755 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:06:01.756 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2023-09-30T01:06:01.757 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = "172.21.15.79,[v2:172.21.15.79:3301,v1:172.21.15.79:6790],[v2:172.21.15.79:3302,v1:172.21.15.79:6791]" 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: mon client directed command retry = 5 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: mon election default strategy = 3 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: ms bind msgr1 = False 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: ms bind msgr2 = True 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject socket failures = 5000 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: ms type = async 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 06bf7746-ed30-4322-b1bc-267eae5b6611 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.758 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 600 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2023-09-30T01:06:01.759 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2023-09-30T01:06:01.760 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression algorithm = lz4 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression mode = aggressive 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore zero block detection = True 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 20 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 20 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 10 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2023-09-30T01:06:01.761 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify cached snaps = True 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify missing on start = True 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock override recovery settings = True 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue = debug_random 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue cut off = debug_random 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2023-09-30T01:06:01.762 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2023-09-30T01:06:01.763 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:06:01.764 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: mon scrub interval = 300 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:06:01.765 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:06:01.766 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2023-09-30T01:06:01.766 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.b] 2023-09-30T01:06:01.766 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.c] 2023-09-30T01:06:01.767 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T01:06:01.767 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T01:06:01.860 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T01:06:01.860 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:01.860 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T01:06:01.879 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:01.879 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T01:06:01.937 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:06:01.937 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:01.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:06:01.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:06:02.014 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:02.014 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:02.072 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T01:06:02.072 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T01:06:02.072 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:06:02.162 INFO:teuthology.orchestra.run.smithi079.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:06:02.164 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T01:06:02.165 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T01:06:02.165 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T01:06:02.205 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T01:06:02.217 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T01:06:02.217 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-30T01:06:02.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T01:06:02.284 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:06:02.284 INFO:tasks.ceph:role: osd.0 2023-09-30T01:06:02.284 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:06:02.284 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:06:02.359 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:06:02.360 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:06:02.360 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:06:02.360 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:06:02.793 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2023-09-30T01:06:02.795 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2023-09-30T01:06:02.795 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T01:06:02.976 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T01:06:03.011 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T01:06:03.079 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:06:03.079 INFO:tasks.ceph:role: osd.1 2023-09-30T01:06:03.079 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:06:03.079 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:06:03.153 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:06:03.154 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:06:03.578 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2023-09-30T01:06:03.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2023-09-30T01:06:03.580 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T01:06:03.692 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T01:06:03.723 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T01:06:03.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:06:03.791 INFO:tasks.ceph:role: osd.2 2023-09-30T01:06:03.791 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:06:03.791 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:06:03.866 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:06:03.866 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:06:03.866 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:06:03.866 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T01:06:03.866 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:06:03.867 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:06:03.867 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:06:03.867 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:06:03.867 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:06:03.867 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:06:04.314 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2023-09-30T01:06:04.316 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2023-09-30T01:06:04.316 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T01:06:04.475 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T01:06:04.507 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:04.600 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:04.600+0000 7f7ca40a4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T01:06:04.603 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:04.603+0000 7f7ca40a4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T01:06:04.603 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:04.603+0000 7f7ca40a4540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:06:04.603 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:04.603+0000 7f7ca40a4540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:06:04.603 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:04.603+0000 7f7ca40a4540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:06:04.608 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:04.608+0000 7f7ca40a4540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T01:06:05.138 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:05.138+0000 7f7ca40a4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:05.896 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:05.896+0000 7f7ca40a4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:06.659 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:06.659+0000 7f7ca40a4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:07.456 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T01:06:07.490 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:07.579 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:07.578+0000 7f399e5a1540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T01:06:07.579 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:07.579+0000 7f399e5a1540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T01:06:07.579 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:07.579+0000 7f399e5a1540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:06:07.579 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:07.579+0000 7f399e5a1540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:06:07.580 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:07.579+0000 7f399e5a1540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:06:07.580 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:07.579+0000 7f399e5a1540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T01:06:08.103 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:08.103+0000 7f399e5a1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:08.869 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:08.869+0000 7f399e5a1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:09.631 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:09.631+0000 7f399e5a1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:10.428 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T01:06:10.461 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:10.552 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:10.551+0000 7f60f65c2540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T01:06:10.552 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:10.551+0000 7f60f65c2540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T01:06:10.552 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:10.551+0000 7f60f65c2540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:06:10.553 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:10.551+0000 7f60f65c2540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:06:10.553 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:10.551+0000 7f60f65c2540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:06:10.553 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:10.552+0000 7f60f65c2540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T01:06:11.078 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:11.077+0000 7f60f65c2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:11.839 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:11.839+0000 7f60f65c2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:12.603 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:12.603+0000 7f60f65c2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:06:13.395 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T01:06:13.428 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T01:06:13.428 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:13.428 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T01:06:13.498 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:13.498 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T01:06:13.568 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:13.568 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T01:06:13.637 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:13.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T01:06:13.706 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:13.706 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T01:06:13.765 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T01:06:13.765 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:06:13.831 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB6dBdlrqbBCRAAmID2IArokqrHCo74vA7Klw== 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB8dBdlcRLaIxAAKOjRswlRjZuseSPFdmNJYA== 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB/dBdlleWQIhAAxxVZYqqjIL2cZkTo5oBZKA== 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQCCdBdl8nTtIBAAq5rS8Gsx99NObuXEk4vZDQ== 2023-09-30T01:06:13.833 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2023-09-30T01:06:13.834 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQB6dBdlC+9NDBAA6aqOqSz5tjODdyrMT8tnKg== 2023-09-30T01:06:13.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:06:13.925 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:06:13.974 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:06:14.024 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:06:14.074 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:06:14.127 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T01:06:14.127 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T01:06:14.154 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:06:14.248 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T01:06:14.275 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T01:06:14.342 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:06:14.437 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T01:06:14.464 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T01:06:14.531 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:06:14.625 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T01:06:14.652 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:06:14.709 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T01:06:14.710 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T01:06:14.710 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T01:06:14.752 INFO:tasks.ceph.mon.a:Started 2023-09-30T01:06:14.752 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T01:06:14.752 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T01:06:14.754 INFO:tasks.ceph.mon.b:Started 2023-09-30T01:06:14.754 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T01:06:14.754 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T01:06:14.756 INFO:tasks.ceph.mon.c:Started 2023-09-30T01:06:14.756 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T01:06:14.756 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T01:06:14.757 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T01:06:14.759 INFO:tasks.ceph.mgr.x:Started 2023-09-30T01:06:14.759 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T01:06:14.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T01:06:21.183 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:21.183+0000 7fe6a9e131c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:06:21.192 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2023-09-30T01:06:21.203 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T01:06:21.204 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:21.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T01:06:21.231 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:21.231 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T01:06:21.310 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2023-09-30T01:06:21.310 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T01:06:21.360 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:21.360+0000 7fe6a9e131c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T01:06:21.377 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 9859f6f1-84cd-4169-83f2-e592bab9ab7a 0 2023-09-30T01:06:21.627 INFO:teuthology.orchestra.run.smithi079.stdout:0 2023-09-30T01:06:21.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new 2acb9b56-ede4-49be-b2be-314942b6107a 1 2023-09-30T01:06:21.712 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:21.712+0000 7fe6a9e131c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:06:21.959 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:21.959+0000 7fcb2affd700 0 --2- 172.21.15.79:0/1832196839 >> [v2:172.21.15.79:3302/0,v1:172.21.15.79:6791/0] conn(0x7fcb2c0692a0 0x7fcb2c069670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-30T01:06:22.049 INFO:teuthology.orchestra.run.smithi079.stdout:1 2023-09-30T01:06:22.062 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd new ac4f74f0-074c-41dc-9b6b-6232b4097a5f 2 2023-09-30T01:06:22.233 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:22.233+0000 7fe6a9e131c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T01:06:22.370 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:22.369+0000 7fe6a9e131c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T01:06:22.384 INFO:teuthology.orchestra.run.smithi079.stderr:2023-09-30T01:06:22.384+0000 7f555a228700 0 --2- 172.21.15.79:0/4076973172 >> [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] conn(0x7f5554069cd0 0x7f5538006150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-30T01:06:22.474 INFO:teuthology.orchestra.run.smithi079.stdout:2 2023-09-30T01:06:22.486 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T01:06:22.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T01:06:22.489 INFO:tasks.ceph.osd.0:Started 2023-09-30T01:06:22.489 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T01:06:22.489 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T01:06:22.492 INFO:tasks.ceph.osd.1:Started 2023-09-30T01:06:22.492 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T01:06:22.492 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T01:06:22.494 INFO:tasks.ceph.osd.2:Started 2023-09-30T01:06:22.495 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:06:22.700 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:22.700 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":5,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:22.471736+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:06:22.713 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T01:06:22.713 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T01:06:22.753 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:22.753+0000 7fe6a9e131c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:06:23.015 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:06:23.098 INFO:tasks.ceph.osd.1.smithi079.stderr:2023-09-30T01:06:23.097+0000 7fbd93cc1540 -1 Falling back to public interface 2023-09-30T01:06:23.103 INFO:tasks.ceph.osd.0.smithi079.stderr:2023-09-30T01:06:23.103+0000 7efcf7c62540 -1 Falling back to public interface 2023-09-30T01:06:23.111 INFO:tasks.ceph.osd.2.smithi079.stderr:2023-09-30T01:06:23.111+0000 7f66eaa0d540 -1 Falling back to public interface 2023-09-30T01:06:23.200 INFO:teuthology.misc.health.smithi079.stdout: 2023-09-30T01:06:23.201 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":5,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:22.471736+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:06:23.212 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-30T01:06:23.486 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:23.486+0000 7fe6a9e131c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:06:23.556 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:23.556+0000 7fe6a9e131c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:06:23.689 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:23.689+0000 7fe6a9e131c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:06:23.756 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:23.756+0000 7fe6a9e131c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:06:23.932 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:23.932+0000 7fe6a9e131c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:06:24.008 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.008+0000 7fe6a9e131c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:06:24.083 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.083+0000 7fe6a9e131c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:06:24.178 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.178+0000 7fe6a9e131c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:06:24.357 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.357+0000 7fe6a9e131c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:06:24.435 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.435+0000 7fe6a9e131c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:06:24.719 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.718+0000 7fe6a9e131c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:06:24.789 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:24.788+0000 7fe6a9e131c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:06:25.288 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:25.288+0000 7fe6a9e131c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:06:25.606 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:25.606+0000 7fe6a9e131c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:06:25.745 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:25.745+0000 7fe6a9e131c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:06:26.141 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.141+0000 7fe6a9e131c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:06:26.285 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.285+0000 7fe6a9e131c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:06:26.437 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.437+0000 7fe6a9e131c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:06:26.535 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.535+0000 7fe6a9e131c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:06:26.583 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.583+0000 7fe6392bd700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:06:26.583 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.583+0000 7fe6392bd700 -1 pg_autoscaler.serve: 2023-09-30T01:06:26.584 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:26.583+0000 7fe6392bd700 -1 Traceback (most recent call last): 2023-09-30T01:06:26.584 INFO:tasks.ceph.mgr.x.smithi079.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T01:06:26.584 INFO:tasks.ceph.mgr.x.smithi079.stderr: self._maybe_adjust() 2023-09-30T01:06:26.584 INFO:tasks.ceph.mgr.x.smithi079.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T01:06:26.584 INFO:tasks.ceph.mgr.x.smithi079.stderr: if self.noautoscale: 2023-09-30T01:06:26.585 INFO:tasks.ceph.mgr.x.smithi079.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:06:26.585 INFO:tasks.ceph.mgr.x.smithi079.stderr: 2023-09-30T01:06:26.657 INFO:tasks.ceph.osd.1.smithi079.stderr:2023-09-30T01:06:26.656+0000 7fbd93cc1540 -1 osd.1 0 log_to_monitors true 2023-09-30T01:06:26.666 INFO:tasks.ceph.osd.2.smithi079.stderr:2023-09-30T01:06:26.666+0000 7f66eaa0d540 -1 osd.2 0 log_to_monitors true 2023-09-30T01:06:26.678 INFO:tasks.ceph.osd.0.smithi079.stderr:2023-09-30T01:06:26.677+0000 7efcf7c62540 -1 osd.0 0 log_to_monitors true 2023-09-30T01:06:27.563 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:27.563+0000 7fe66e4f7700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T01:06:27.564 INFO:tasks.ceph.mgr.x.smithi079.stderr:2023-09-30T01:06:27.564+0000 7fe66e4f7700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T01:06:28.633 INFO:tasks.ceph.osd.1.smithi079.stderr:2023-09-30T01:06:28.633+0000 7fbd8a8c6700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T01:06:28.637 INFO:tasks.ceph.osd.1.smithi079.stderr:2023-09-30T01:06:28.637+0000 7fbd866c3700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:06:28.640 INFO:tasks.ceph.osd.2.smithi079.stderr:2023-09-30T01:06:28.640+0000 7f66e1612700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T01:06:28.640 INFO:tasks.ceph.osd.0.smithi079.stderr:2023-09-30T01:06:28.640+0000 7efcee867700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T01:06:28.643 INFO:tasks.ceph.osd.2.smithi079.stderr:2023-09-30T01:06:28.643+0000 7f66dd40f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:06:28.644 INFO:tasks.ceph.osd.0.smithi079.stderr:2023-09-30T01:06:28.644+0000 7efcea664700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:06:29.514 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:06:29.831 INFO:teuthology.misc.health.smithi079.stdout: 2023-09-30T01:06:29.831 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":8,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:29.635105+0000","last_up_change":"2023-09-30T01:06:29.635105+0000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":114631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":114631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":114631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":114631}]},"public_addr":"172.21.15.79:6804/114631","cluster_addr":"172.21.15.79:6805/114631","heartbeat_back_addr":"172.21.15.79:6807/114631","heartbeat_front_addr":"172.21.15.79:6806/114631","state":["exists","up"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":114632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":114632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":114632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":114632}]},"public_addr":"172.21.15.79:6800/114632","cluster_addr":"172.21.15.79:6801/114632","heartbeat_back_addr":"172.21.15.79:6803/114632","heartbeat_front_addr":"172.21.15.79:6802/114632","state":["exists","up"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":114663}]},"public_addr":"172.21.15.79:6808/114663","cluster_addr":"172.21.15.79:6809/114663","heartbeat_back_addr":"172.21.15.79:6811/114663","heartbeat_front_addr":"172.21.15.79:6810/114663","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:06:29.844 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-30T01:06:29.844 INFO:tasks.ceph:Creating RBD pool 2023-09-30T01:06:29.844 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T01:06:30.637 INFO:tasks.ceph.mon.a.smithi079.stderr:2023-09-30T01:06:30.637+0000 7faad9423700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:06:30.650 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'rbd' created 2023-09-30T01:06:30.662 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T01:06:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T01:06:31.674 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T01:06:31.674 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T01:06:31.674 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T01:06:31.675 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T01:06:31.675 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T01:06:32.079 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:32.094 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":114135}]},"active_addr":"172.21.15.79:6812/114135","active_change":"2023-09-30T01:06:26.549649+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2497444743}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":1368064888}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3554260344}]}]} 2023-09-30T01:06:32.096 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T01:06:32.096 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T01:06:32.097 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:06:32.414 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":10,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:31.645338+0000","last_up_change":"2023-09-30T01:06:29.635105+0000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:06:30.168858+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:06:30.600035+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":114631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":114631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":114631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":114631}]},"public_addr":"172.21.15.79:6804/114631","cluster_addr":"172.21.15.79:6805/114631","heartbeat_back_addr":"172.21.15.79:6807/114631","heartbeat_front_addr":"172.21.15.79:6806/114631","state":["exists","up"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":114632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":114632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":114632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":114632}]},"public_addr":"172.21.15.79:6800/114632","cluster_addr":"172.21.15.79:6801/114632","heartbeat_back_addr":"172.21.15.79:6803/114632","heartbeat_front_addr":"172.21.15.79:6802/114632","state":["exists","up"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":114663}]},"public_addr":"172.21.15.79:6808/114663","cluster_addr":"172.21.15.79:6809/114663","heartbeat_back_addr":"172.21.15.79:6811/114663","heartbeat_front_addr":"172.21.15.79:6810/114663","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.650231+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.686193+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.665835+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:06:32.427 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T01:06:32.427 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:06:32.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:32.744 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":11,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:32.650405+0000","last_up_change":"2023-09-30T01:06:29.635105+0000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:06:30.168858+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:06:30.600035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":114631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":114631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":114631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":114631}]},"public_addr":"172.21.15.79:6804/114631","cluster_addr":"172.21.15.79:6805/114631","heartbeat_back_addr":"172.21.15.79:6807/114631","heartbeat_front_addr":"172.21.15.79:6806/114631","state":["exists","up"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":114632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":114632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":114632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":114632}]},"public_addr":"172.21.15.79:6800/114632","cluster_addr":"172.21.15.79:6801/114632","heartbeat_back_addr":"172.21.15.79:6803/114632","heartbeat_front_addr":"172.21.15.79:6802/114632","state":["exists","up"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":114663}]},"public_addr":"172.21.15.79:6808/114663","cluster_addr":"172.21.15.79:6809/114663","heartbeat_back_addr":"172.21.15.79:6811/114663","heartbeat_front_addr":"172.21.15.79:6810/114663","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.650231+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.686193+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.665835+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:06:32.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T01:06:32.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T01:06:32.759 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T01:06:32.927 INFO:teuthology.orchestra.run.smithi079.stdout:34359738371 2023-09-30T01:06:32.928 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:06:32.929 INFO:teuthology.orchestra.run.smithi079.stdout:34359738371 2023-09-30T01:06:32.929 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:06:32.938 INFO:teuthology.orchestra.run.smithi079.stdout:34359738371 2023-09-30T01:06:32.939 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:06:33.262 INFO:teuthology.orchestra.run.smithi079.stdout:34359738370 2023-09-30T01:06:33.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-09-30T01:06:33.304 INFO:teuthology.orchestra.run.smithi079.stdout:34359738370 2023-09-30T01:06:33.314 INFO:teuthology.orchestra.run.smithi079.stdout:34359738370 2023-09-30T01:06:33.318 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2023-09-30T01:06:33.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2023-09-30T01:06:34.279 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:06:34.319 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:06:34.328 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:06:34.619 INFO:teuthology.orchestra.run.smithi079.stdout:34359738370 2023-09-30T01:06:34.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-09-30T01:06:34.663 INFO:teuthology.orchestra.run.smithi079.stdout:34359738371 2023-09-30T01:06:34.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2023-09-30T01:06:34.678 DEBUG:teuthology.parallel:result is None 2023-09-30T01:06:34.702 INFO:teuthology.orchestra.run.smithi079.stdout:34359738371 2023-09-30T01:06:34.716 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.0 2023-09-30T01:06:34.716 DEBUG:teuthology.parallel:result is None 2023-09-30T01:06:35.634 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:06:35.960 INFO:teuthology.orchestra.run.smithi079.stdout:34359738371 2023-09-30T01:06:35.974 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2023-09-30T01:06:35.974 DEBUG:teuthology.parallel:result is None 2023-09-30T01:06:35.974 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:06:35.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:06:36.278 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:36.280 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:06:36.292 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-09-30T01:06:34.559441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16008,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":15616,"kb_avail":283099512,"statfs":{"total":289910292480,"available":289893900288,"internally_reserved":0,"allocated":376832,"data_stored":999656,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15990784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664042+0000","last_change":"2023-09-30T01:06:31.662640+0000","last_active":"2023-09-30T01:06:31.664042+0000","last_peered":"2023-09-30T01:06:31.664042+0000","last_clean":"2023-09-30T01:06:31.664042+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:06:31.664042+0000","last_undegraded":"2023-09-30T01:06:31.664042+0000","last_fullsized":"2023-09-30T01:06:31.664042+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:16:33.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666733+0000","last_change":"2023-09-30T01:06:31.665194+0000","last_active":"2023-09-30T01:06:31.666733+0000","last_peered":"2023-09-30T01:06:31.666733+0000","last_clean":"2023-09-30T01:06:31.666733+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:06:31.666733+0000","last_undegraded":"2023-09-30T01:06:31.666733+0000","last_fullsized":"2023-09-30T01:06:31.666733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:57:04.234397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666542+0000","last_change":"2023-09-30T01:06:31.664895+0000","last_active":"2023-09-30T01:06:31.666542+0000","last_peered":"2023-09-30T01:06:31.666542+0000","last_clean":"2023-09-30T01:06:31.666542+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:06:31.666542+0000","last_undegraded":"2023-09-30T01:06:31.666542+0000","last_fullsized":"2023-09-30T01:06:31.666542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:23.197758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661339+0000","last_change":"2023-09-30T01:06:31.660865+0000","last_active":"2023-09-30T01:06:31.661339+0000","last_peered":"2023-09-30T01:06:31.661339+0000","last_clean":"2023-09-30T01:06:31.661339+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:06:31.661339+0000","last_undegraded":"2023-09-30T01:06:31.661339+0000","last_fullsized":"2023-09-30T01:06:31.661339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:05.331355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.658586+0000","last_change":"2023-09-30T01:06:31.658586+0000","last_active":"2023-09-30T01:06:31.658586+0000","last_peered":"2023-09-30T01:06:31.658586+0000","last_clean":"2023-09-30T01:06:31.658586+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:06:31.658586+0000","last_undegraded":"2023-09-30T01:06:31.658586+0000","last_fullsized":"2023-09-30T01:06:31.658586+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:14:18.689341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:32.234809+0000","last_change":"2023-09-30T01:06:31.663585+0000","last_active":"2023-09-30T01:06:32.234809+0000","last_peered":"2023-09-30T01:06:32.234809+0000","last_clean":"2023-09-30T01:06:32.234809+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:06:32.234809+0000","last_undegraded":"2023-09-30T01:06:32.234809+0000","last_fullsized":"2023-09-30T01:06:32.234809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:00.773713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664060+0000","last_change":"2023-09-30T01:06:31.662676+0000","last_active":"2023-09-30T01:06:31.664060+0000","last_peered":"2023-09-30T01:06:31.664060+0000","last_clean":"2023-09-30T01:06:31.664060+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:06:31.664060+0000","last_undegraded":"2023-09-30T01:06:31.664060+0000","last_fullsized":"2023-09-30T01:06:31.664060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:44.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661334+0000","last_change":"2023-09-30T01:06:31.660666+0000","last_active":"2023-09-30T01:06:31.661334+0000","last_peered":"2023-09-30T01:06:31.661334+0000","last_clean":"2023-09-30T01:06:31.661334+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:06:31.661334+0000","last_undegraded":"2023-09-30T01:06:31.661334+0000","last_fullsized":"2023-09-30T01:06:31.661334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:58:38.509668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.669292+0000","last_change":"2023-09-30T01:06:31.665693+0000","last_active":"2023-09-30T01:06:31.669292+0000","last_peered":"2023-09-30T01:06:31.669292+0000","last_clean":"2023-09-30T01:06:31.669292+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:06:31.669292+0000","last_undegraded":"2023-09-30T01:06:31.669292+0000","last_fullsized":"2023-09-30T01:06:31.669292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:10.320073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.189}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":90112,"data_stored":20776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.186}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5456,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366384,"statfs":{"total":96636764160,"available":96631177216,"internally_reserved":0,"allocated":139264,"data_stored":487732,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:06:36.293 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:06:36.598 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:36.601 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:06:36.612 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-09-30T01:06:36.559849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16016,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":15616,"kb_avail":283099504,"statfs":{"total":289910292480,"available":289893892096,"internally_reserved":0,"allocated":385024,"data_stored":1003072,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15990784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000408"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664042+0000","last_change":"2023-09-30T01:06:31.662640+0000","last_active":"2023-09-30T01:06:31.664042+0000","last_peered":"2023-09-30T01:06:31.664042+0000","last_clean":"2023-09-30T01:06:31.664042+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:06:31.664042+0000","last_undegraded":"2023-09-30T01:06:31.664042+0000","last_fullsized":"2023-09-30T01:06:31.664042+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:16:33.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666733+0000","last_change":"2023-09-30T01:06:31.665194+0000","last_active":"2023-09-30T01:06:31.666733+0000","last_peered":"2023-09-30T01:06:31.666733+0000","last_clean":"2023-09-30T01:06:31.666733+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:06:31.666733+0000","last_undegraded":"2023-09-30T01:06:31.666733+0000","last_fullsized":"2023-09-30T01:06:31.666733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:57:04.234397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666542+0000","last_change":"2023-09-30T01:06:31.664895+0000","last_active":"2023-09-30T01:06:31.666542+0000","last_peered":"2023-09-30T01:06:31.666542+0000","last_clean":"2023-09-30T01:06:31.666542+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:06:31.666542+0000","last_undegraded":"2023-09-30T01:06:31.666542+0000","last_fullsized":"2023-09-30T01:06:31.666542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:23.197758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661339+0000","last_change":"2023-09-30T01:06:31.660865+0000","last_active":"2023-09-30T01:06:31.661339+0000","last_peered":"2023-09-30T01:06:31.661339+0000","last_clean":"2023-09-30T01:06:31.661339+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:06:31.661339+0000","last_undegraded":"2023-09-30T01:06:31.661339+0000","last_fullsized":"2023-09-30T01:06:31.661339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:05.331355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.658586+0000","last_change":"2023-09-30T01:06:31.658586+0000","last_active":"2023-09-30T01:06:31.658586+0000","last_peered":"2023-09-30T01:06:31.658586+0000","last_clean":"2023-09-30T01:06:31.658586+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:06:31.658586+0000","last_undegraded":"2023-09-30T01:06:31.658586+0000","last_fullsized":"2023-09-30T01:06:31.658586+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:14:18.689341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:32.234809+0000","last_change":"2023-09-30T01:06:31.663585+0000","last_active":"2023-09-30T01:06:32.234809+0000","last_peered":"2023-09-30T01:06:32.234809+0000","last_clean":"2023-09-30T01:06:32.234809+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:06:32.234809+0000","last_undegraded":"2023-09-30T01:06:32.234809+0000","last_fullsized":"2023-09-30T01:06:32.234809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:00.773713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664060+0000","last_change":"2023-09-30T01:06:31.662676+0000","last_active":"2023-09-30T01:06:31.664060+0000","last_peered":"2023-09-30T01:06:31.664060+0000","last_clean":"2023-09-30T01:06:31.664060+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:06:31.664060+0000","last_undegraded":"2023-09-30T01:06:31.664060+0000","last_fullsized":"2023-09-30T01:06:31.664060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:44.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661334+0000","last_change":"2023-09-30T01:06:31.660666+0000","last_active":"2023-09-30T01:06:31.661334+0000","last_peered":"2023-09-30T01:06:31.661334+0000","last_clean":"2023-09-30T01:06:31.661334+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:06:31.661334+0000","last_undegraded":"2023-09-30T01:06:31.661334+0000","last_fullsized":"2023-09-30T01:06:31.661334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:58:38.509668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.669292+0000","last_change":"2023-09-30T01:06:31.665693+0000","last_active":"2023-09-30T01:06:31.669292+0000","last_peered":"2023-09-30T01:06:31.669292+0000","last_clean":"2023-09-30T01:06:31.669292+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:06:31.669292+0000","last_undegraded":"2023-09-30T01:06:31.669292+0000","last_fullsized":"2023-09-30T01:06:31.669292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:10.320073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.189}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5088,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366752,"statfs":{"total":96636764160,"available":96631554048,"internally_reserved":0,"allocated":90112,"data_stored":20776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.186}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:06:36.613 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:06:36.613 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T01:06:36.613 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T01:06:36.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:36.963 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:36.964 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:39.978 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:40.338 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:40.339 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:43.353 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:43.714 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:43.714 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:46.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:47.092 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:47.092 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:50.106 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:50.466 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:50.466 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:53.480 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:53.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:53.845 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:06:56.860 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:06:57.220 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:06:57.221 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:00.235 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:00.594 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:00.595 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:03.609 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:03.971 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:03.971 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:06.985 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:07.342 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:07.342 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:10.356 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:10.720 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:10.720 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:13.734 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:14.095 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:14.095 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:17.110 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:17.468 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:17.469 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:20.483 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:20.848 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:20.848 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:23.862 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:24.223 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:24.223 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:27.238 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:27.599 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:27.599 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:30.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:30.976 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:30.976 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:33.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:34.350 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:34.351 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:37.365 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:37.727 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:37.727 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:40.744 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:41.104 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:41.104 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:44.119 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:44.478 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:44.478 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:47.493 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:47.854 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:47.855 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:50.869 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:51.233 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:51.233 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:54.247 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:54.611 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:54.611 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:07:57.625 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:07:57.984 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:07:57.984 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:00.998 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:01.359 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:01.359 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:04.373 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:04.735 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:04.735 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:07.750 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:08.113 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:08.113 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:11.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:11.488 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:11.488 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:14.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:14.864 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:14.864 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:17.879 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:18.238 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:18.238 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:21.253 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:21.613 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:21.614 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:24.629 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:24.992 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:24.992 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:28.007 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:28.365 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:28.365 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:31.380 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:31.741 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:31.741 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:34.755 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:35.116 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:35.116 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:38.130 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:38.488 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:38.488 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:41.503 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:41.867 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:41.867 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:44.881 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:45.242 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:45.243 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:48.257 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:48.615 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:48.615 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:51.629 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:51.995 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:51.995 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:55.009 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:55.371 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:55.372 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:08:58.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:08:58.749 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:08:58.749 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:01.764 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:02.126 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:02.126 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:05.140 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:05.499 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:05.499 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:08.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:08.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:08.872 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:11.886 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:12.245 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:12.246 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:15.260 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:15.621 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:15.621 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:18.635 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:18.995 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:18.996 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:22.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:22.371 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:22.371 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:25.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:25.749 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:25.749 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:28.763 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:29.126 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:29.126 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:32.140 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:32.498 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:32.498 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:35.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:35.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:35.872 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:38.886 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:39.247 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:39.247 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:42.261 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:42.622 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:42.622 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:45.636 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:45.993 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:45.993 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:49.008 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:49.367 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:49.367 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:52.381 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:52.741 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:52.741 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:55.755 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:56.117 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:56.118 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:59.132 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:09:59.492 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:09:59.492 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:09:59.999 INFO:tasks.ceph.mon.a.smithi079.stderr:2023-09-30T01:09:59.999+0000 7faadbc28700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:10:02.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:02.867 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:02.867 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:05.881 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:06.249 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:06.250 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:09.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:09.626 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:09.626 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:12.640 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:12.999 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:12.999 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:16.014 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:16.372 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:16.372 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:19.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:19.746 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:19.747 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:22.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:23.120 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:23.120 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:26.135 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:26.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:26.497 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:29.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:29.871 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:29.871 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:32.885 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:33.248 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:33.248 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:36.262 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:36.622 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:36.622 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:39.634 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:39.998 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:39.999 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:43.013 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:43.374 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:43.374 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:46.388 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:46.748 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:46.748 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:49.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:50.124 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:50.124 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:53.138 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:53.497 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:53.498 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:56.512 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:10:56.871 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:10:56.871 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:10:59.885 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:00.248 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:00.248 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:03.262 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:03.623 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:06.637 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:06.997 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:06.997 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:10.012 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:10.372 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:10.372 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:13.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:13.747 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:13.747 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:16.761 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:17.125 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:17.125 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:20.139 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:20.497 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:20.497 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:23.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:23.868 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:23.868 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:26.883 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:27.242 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:27.242 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:30.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:30.614 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:30.615 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:33.629 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:33.990 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:33.991 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:37.005 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:11:37.365 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:37.365 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:11:37.378 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T01:11:37.379 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:11:37.689 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:37.689 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":11,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:32.650405+0000","last_up_change":"2023-09-30T01:06:29.635105+0000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:06:30.168858+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:06:30.600035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":114631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":114631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":114631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":114631}]},"public_addr":"172.21.15.79:6804/114631","cluster_addr":"172.21.15.79:6805/114631","heartbeat_back_addr":"172.21.15.79:6807/114631","heartbeat_front_addr":"172.21.15.79:6806/114631","state":["exists","up"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":114632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":114632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":114632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":114632}]},"public_addr":"172.21.15.79:6800/114632","cluster_addr":"172.21.15.79:6801/114632","heartbeat_back_addr":"172.21.15.79:6803/114632","heartbeat_front_addr":"172.21.15.79:6802/114632","state":["exists","up"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":114663}]},"public_addr":"172.21.15.79:6808/114663","cluster_addr":"172.21.15.79:6809/114663","heartbeat_back_addr":"172.21.15.79:6811/114663","heartbeat_front_addr":"172.21.15.79:6810/114663","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.650231+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.686193+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.665835+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:11:37.702 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:11:37.702 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:11:38.009 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:38.011 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:11:38.022 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2023-09-30T01:11:36.619699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16024,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":15616,"kb_avail":283099496,"statfs":{"total":289910292480,"available":289893883904,"internally_reserved":0,"allocated":393216,"data_stored":1006488,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15990784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002243"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664042+0000","last_change":"2023-09-30T01:06:31.662640+0000","last_active":"2023-09-30T01:06:31.664042+0000","last_peered":"2023-09-30T01:06:31.664042+0000","last_clean":"2023-09-30T01:06:31.664042+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:06:31.664042+0000","last_undegraded":"2023-09-30T01:06:31.664042+0000","last_fullsized":"2023-09-30T01:06:31.664042+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:16:33.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666733+0000","last_change":"2023-09-30T01:06:31.665194+0000","last_active":"2023-09-30T01:06:31.666733+0000","last_peered":"2023-09-30T01:06:31.666733+0000","last_clean":"2023-09-30T01:06:31.666733+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:06:31.666733+0000","last_undegraded":"2023-09-30T01:06:31.666733+0000","last_fullsized":"2023-09-30T01:06:31.666733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:57:04.234397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666542+0000","last_change":"2023-09-30T01:06:31.664895+0000","last_active":"2023-09-30T01:06:31.666542+0000","last_peered":"2023-09-30T01:06:31.666542+0000","last_clean":"2023-09-30T01:06:31.666542+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:06:31.666542+0000","last_undegraded":"2023-09-30T01:06:31.666542+0000","last_fullsized":"2023-09-30T01:06:31.666542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:23.197758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661339+0000","last_change":"2023-09-30T01:06:31.660865+0000","last_active":"2023-09-30T01:06:31.661339+0000","last_peered":"2023-09-30T01:06:31.661339+0000","last_clean":"2023-09-30T01:06:31.661339+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:06:31.661339+0000","last_undegraded":"2023-09-30T01:06:31.661339+0000","last_fullsized":"2023-09-30T01:06:31.661339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:05.331355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664420+0000","last_change":"2023-09-30T01:06:31.658586+0000","last_active":"2023-09-30T01:06:31.664420+0000","last_peered":"2023-09-30T01:06:31.664420+0000","last_clean":"2023-09-30T01:06:31.664420+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:06:31.664420+0000","last_undegraded":"2023-09-30T01:06:31.664420+0000","last_fullsized":"2023-09-30T01:06:31.664420+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:14:18.689341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:32.234809+0000","last_change":"2023-09-30T01:06:31.663585+0000","last_active":"2023-09-30T01:06:32.234809+0000","last_peered":"2023-09-30T01:06:32.234809+0000","last_clean":"2023-09-30T01:06:32.234809+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:06:32.234809+0000","last_undegraded":"2023-09-30T01:06:32.234809+0000","last_fullsized":"2023-09-30T01:06:32.234809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:00.773713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664060+0000","last_change":"2023-09-30T01:06:31.662676+0000","last_active":"2023-09-30T01:06:31.664060+0000","last_peered":"2023-09-30T01:06:31.664060+0000","last_clean":"2023-09-30T01:06:31.664060+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:06:31.664060+0000","last_undegraded":"2023-09-30T01:06:31.664060+0000","last_fullsized":"2023-09-30T01:06:31.664060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:44.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661334+0000","last_change":"2023-09-30T01:06:31.660666+0000","last_active":"2023-09-30T01:06:31.661334+0000","last_peered":"2023-09-30T01:06:31.661334+0000","last_clean":"2023-09-30T01:06:31.661334+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:06:31.661334+0000","last_undegraded":"2023-09-30T01:06:31.661334+0000","last_fullsized":"2023-09-30T01:06:31.661334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:58:38.509668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.669292+0000","last_change":"2023-09-30T01:06:31.665693+0000","last_active":"2023-09-30T01:06:31.669292+0000","last_peered":"2023-09-30T01:06:31.669292+0000","last_clean":"2023-09-30T01:06:31.669292+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:06:31.669292+0000","last_undegraded":"2023-09-30T01:06:31.669292+0000","last_fullsized":"2023-09-30T01:06:31.669292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:10.320073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.216,"5min":0.189,"15min":0.189},"max":{"1min":0.42999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.33100000000000002,"15min":0.33300000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34000000000000002,"15min":0.35099999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.436,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.255},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.22900000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.46200000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366744,"statfs":{"total":96636764160,"available":96631545856,"internally_reserved":0,"allocated":98304,"data_stored":24192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29399999999999998},"min":{"1min":0.23499999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.42899999999999999,"5min":0.498,"15min":0.498},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"min":{"1min":0.25900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.48199999999999998,"5min":0.503,"15min":0.503},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.30599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.312,"15min":0.31900000000000001},"min":{"1min":0.189,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40400000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30599999999999999,"15min":0.30299999999999999},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.33400000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.44600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.373},{"interface":"front","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45400000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:11:38.023 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:11:38.329 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:38.331 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:11:38.342 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2023-09-30T01:11:36.619699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16024,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":15616,"kb_avail":283099496,"statfs":{"total":289910292480,"available":289893883904,"internally_reserved":0,"allocated":393216,"data_stored":1006488,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15990784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002243"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664042+0000","last_change":"2023-09-30T01:06:31.662640+0000","last_active":"2023-09-30T01:06:31.664042+0000","last_peered":"2023-09-30T01:06:31.664042+0000","last_clean":"2023-09-30T01:06:31.664042+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:06:31.664042+0000","last_undegraded":"2023-09-30T01:06:31.664042+0000","last_fullsized":"2023-09-30T01:06:31.664042+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:16:33.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666733+0000","last_change":"2023-09-30T01:06:31.665194+0000","last_active":"2023-09-30T01:06:31.666733+0000","last_peered":"2023-09-30T01:06:31.666733+0000","last_clean":"2023-09-30T01:06:31.666733+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:06:31.666733+0000","last_undegraded":"2023-09-30T01:06:31.666733+0000","last_fullsized":"2023-09-30T01:06:31.666733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:57:04.234397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666542+0000","last_change":"2023-09-30T01:06:31.664895+0000","last_active":"2023-09-30T01:06:31.666542+0000","last_peered":"2023-09-30T01:06:31.666542+0000","last_clean":"2023-09-30T01:06:31.666542+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:06:31.666542+0000","last_undegraded":"2023-09-30T01:06:31.666542+0000","last_fullsized":"2023-09-30T01:06:31.666542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:23.197758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661339+0000","last_change":"2023-09-30T01:06:31.660865+0000","last_active":"2023-09-30T01:06:31.661339+0000","last_peered":"2023-09-30T01:06:31.661339+0000","last_clean":"2023-09-30T01:06:31.661339+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:06:31.661339+0000","last_undegraded":"2023-09-30T01:06:31.661339+0000","last_fullsized":"2023-09-30T01:06:31.661339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:05.331355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664420+0000","last_change":"2023-09-30T01:06:31.658586+0000","last_active":"2023-09-30T01:06:31.664420+0000","last_peered":"2023-09-30T01:06:31.664420+0000","last_clean":"2023-09-30T01:06:31.664420+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:06:31.664420+0000","last_undegraded":"2023-09-30T01:06:31.664420+0000","last_fullsized":"2023-09-30T01:06:31.664420+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:14:18.689341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:32.234809+0000","last_change":"2023-09-30T01:06:31.663585+0000","last_active":"2023-09-30T01:06:32.234809+0000","last_peered":"2023-09-30T01:06:32.234809+0000","last_clean":"2023-09-30T01:06:32.234809+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:06:32.234809+0000","last_undegraded":"2023-09-30T01:06:32.234809+0000","last_fullsized":"2023-09-30T01:06:32.234809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:00.773713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664060+0000","last_change":"2023-09-30T01:06:31.662676+0000","last_active":"2023-09-30T01:06:31.664060+0000","last_peered":"2023-09-30T01:06:31.664060+0000","last_clean":"2023-09-30T01:06:31.664060+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:06:31.664060+0000","last_undegraded":"2023-09-30T01:06:31.664060+0000","last_fullsized":"2023-09-30T01:06:31.664060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:44.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661334+0000","last_change":"2023-09-30T01:06:31.660666+0000","last_active":"2023-09-30T01:06:31.661334+0000","last_peered":"2023-09-30T01:06:31.661334+0000","last_clean":"2023-09-30T01:06:31.661334+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:06:31.661334+0000","last_undegraded":"2023-09-30T01:06:31.661334+0000","last_fullsized":"2023-09-30T01:06:31.661334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:58:38.509668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.669292+0000","last_change":"2023-09-30T01:06:31.665693+0000","last_active":"2023-09-30T01:06:31.669292+0000","last_peered":"2023-09-30T01:06:31.669292+0000","last_clean":"2023-09-30T01:06:31.669292+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:06:31.669292+0000","last_undegraded":"2023-09-30T01:06:31.669292+0000","last_fullsized":"2023-09-30T01:06:31.669292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:10.320073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.216,"5min":0.189,"15min":0.189},"max":{"1min":0.42999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.33100000000000002,"15min":0.33300000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34000000000000002,"15min":0.35099999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.436,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.255},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.22900000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.46200000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366744,"statfs":{"total":96636764160,"available":96631545856,"internally_reserved":0,"allocated":98304,"data_stored":24192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29399999999999998},"min":{"1min":0.23499999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.42899999999999999,"5min":0.498,"15min":0.498},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"min":{"1min":0.25900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.48199999999999998,"5min":0.503,"15min":0.503},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.30599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.312,"15min":0.31900000000000001},"min":{"1min":0.189,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40400000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30599999999999999,"15min":0.30299999999999999},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.33400000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.44600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.373},{"interface":"front","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45400000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:11:38.343 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:11:38.343 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:11:38.650 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:38.652 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:11:38.663 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2023-09-30T01:11:38.620186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16024,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":15616,"kb_avail":283099496,"statfs":{"total":289910292480,"available":289893883904,"internally_reserved":0,"allocated":393216,"data_stored":1006488,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15990784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664042+0000","last_change":"2023-09-30T01:06:31.662640+0000","last_active":"2023-09-30T01:06:31.664042+0000","last_peered":"2023-09-30T01:06:31.664042+0000","last_clean":"2023-09-30T01:06:31.664042+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:06:31.664042+0000","last_undegraded":"2023-09-30T01:06:31.664042+0000","last_fullsized":"2023-09-30T01:06:31.664042+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:16:33.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666733+0000","last_change":"2023-09-30T01:06:31.665194+0000","last_active":"2023-09-30T01:06:31.666733+0000","last_peered":"2023-09-30T01:06:31.666733+0000","last_clean":"2023-09-30T01:06:31.666733+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:06:31.666733+0000","last_undegraded":"2023-09-30T01:06:31.666733+0000","last_fullsized":"2023-09-30T01:06:31.666733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:57:04.234397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666542+0000","last_change":"2023-09-30T01:06:31.664895+0000","last_active":"2023-09-30T01:06:31.666542+0000","last_peered":"2023-09-30T01:06:31.666542+0000","last_clean":"2023-09-30T01:06:31.666542+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:06:31.666542+0000","last_undegraded":"2023-09-30T01:06:31.666542+0000","last_fullsized":"2023-09-30T01:06:31.666542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:23.197758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661339+0000","last_change":"2023-09-30T01:06:31.660865+0000","last_active":"2023-09-30T01:06:31.661339+0000","last_peered":"2023-09-30T01:06:31.661339+0000","last_clean":"2023-09-30T01:06:31.661339+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:06:31.661339+0000","last_undegraded":"2023-09-30T01:06:31.661339+0000","last_fullsized":"2023-09-30T01:06:31.661339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:05.331355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664420+0000","last_change":"2023-09-30T01:06:31.658586+0000","last_active":"2023-09-30T01:06:31.664420+0000","last_peered":"2023-09-30T01:06:31.664420+0000","last_clean":"2023-09-30T01:06:31.664420+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:06:31.664420+0000","last_undegraded":"2023-09-30T01:06:31.664420+0000","last_fullsized":"2023-09-30T01:06:31.664420+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:14:18.689341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:32.234809+0000","last_change":"2023-09-30T01:06:31.663585+0000","last_active":"2023-09-30T01:06:32.234809+0000","last_peered":"2023-09-30T01:06:32.234809+0000","last_clean":"2023-09-30T01:06:32.234809+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:06:32.234809+0000","last_undegraded":"2023-09-30T01:06:32.234809+0000","last_fullsized":"2023-09-30T01:06:32.234809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:00.773713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664060+0000","last_change":"2023-09-30T01:06:31.662676+0000","last_active":"2023-09-30T01:06:31.664060+0000","last_peered":"2023-09-30T01:06:31.664060+0000","last_clean":"2023-09-30T01:06:31.664060+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:06:31.664060+0000","last_undegraded":"2023-09-30T01:06:31.664060+0000","last_fullsized":"2023-09-30T01:06:31.664060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:44.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661334+0000","last_change":"2023-09-30T01:06:31.660666+0000","last_active":"2023-09-30T01:06:31.661334+0000","last_peered":"2023-09-30T01:06:31.661334+0000","last_clean":"2023-09-30T01:06:31.661334+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:06:31.661334+0000","last_undegraded":"2023-09-30T01:06:31.661334+0000","last_fullsized":"2023-09-30T01:06:31.661334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:58:38.509668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.669292+0000","last_change":"2023-09-30T01:06:31.665693+0000","last_active":"2023-09-30T01:06:31.669292+0000","last_peered":"2023-09-30T01:06:31.669292+0000","last_clean":"2023-09-30T01:06:31.669292+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:06:31.669292+0000","last_undegraded":"2023-09-30T01:06:31.669292+0000","last_fullsized":"2023-09-30T01:06:31.669292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:10.320073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.216,"5min":0.189,"15min":0.189},"max":{"1min":0.42999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.33100000000000002,"15min":0.33300000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34000000000000002,"15min":0.35099999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.436,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.255},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.22900000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.46200000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366744,"statfs":{"total":96636764160,"available":96631545856,"internally_reserved":0,"allocated":98304,"data_stored":24192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29399999999999998},"min":{"1min":0.23499999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.42899999999999999,"5min":0.498,"15min":0.498},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"min":{"1min":0.25900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.48199999999999998,"5min":0.503,"15min":0.503},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.30599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.312,"15min":0.31900000000000001},"min":{"1min":0.189,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40400000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30599999999999999,"15min":0.30299999999999999},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.33400000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.44600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.373},{"interface":"front","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45400000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:11:38.664 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:11:38.971 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:38.971 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":11,"fsid":"06bf7746-ed30-4322-b1bc-267eae5b6611","created":"2023-09-30T01:06:20.912625+0000","modified":"2023-09-30T01:06:32.650405+0000","last_up_change":"2023-09-30T01:06:29.635105+0000","last_in_change":"2023-09-30T01:06:22.471736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T01:06:30.168858+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-30T01:06:30.600035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9859f6f1-84cd-4169-83f2-e592bab9ab7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":114631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6805","nonce":114631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6807","nonce":114631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":114631}]},"public_addr":"172.21.15.79:6804/114631","cluster_addr":"172.21.15.79:6805/114631","heartbeat_back_addr":"172.21.15.79:6807/114631","heartbeat_front_addr":"172.21.15.79:6806/114631","state":["exists","up"]},{"osd":1,"uuid":"2acb9b56-ede4-49be-b2be-314942b6107a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":114632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6801","nonce":114632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6803","nonce":114632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":114632}]},"public_addr":"172.21.15.79:6800/114632","cluster_addr":"172.21.15.79:6801/114632","heartbeat_back_addr":"172.21.15.79:6803/114632","heartbeat_front_addr":"172.21.15.79:6802/114632","state":["exists","up"]},{"osd":2,"uuid":"ac4f74f0-074c-41dc-9b6b-6232b4097a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":114663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6809","nonce":114663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6811","nonce":114663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":114663}]},"public_addr":"172.21.15.79:6808/114663","cluster_addr":"172.21.15.79:6809/114663","heartbeat_back_addr":"172.21.15.79:6811/114663","heartbeat_front_addr":"172.21.15.79:6810/114663","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.650231+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.686193+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:06:27.665835+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:11:39.986 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T01:11:39.986 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:11:40.142 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2023-09-30T01:11:40.142 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:11:40.142 INFO:teuthology.orchestra.run.smithi079.stdout:} 2023-09-30T01:11:40.153 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T01:11:40.462 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T01:11:40.473 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T01:11:40.473 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:11:40.613 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2023-09-30T01:11:40.613 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:11:40.613 INFO:teuthology.orchestra.run.smithi079.stdout:} 2023-09-30T01:11:40.625 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T01:11:40.934 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T01:11:40.945 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T01:11:40.946 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:11:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2023-09-30T01:11:41.087 INFO:teuthology.orchestra.run.smithi079.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:11:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:} 2023-09-30T01:11:41.098 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T01:11:41.409 INFO:teuthology.orchestra.run.smithi079.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T01:11:41.420 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:11:41.724 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:11:41.725 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:11:41.736 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-30T01:11:40.620435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16024,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":15616,"kb_avail":283099496,"statfs":{"total":289910292480,"available":289893883904,"internally_reserved":0,"allocated":393216,"data_stored":1006488,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":15990784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002128"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664042+0000","last_change":"2023-09-30T01:06:31.662640+0000","last_active":"2023-09-30T01:06:31.664042+0000","last_peered":"2023-09-30T01:06:31.664042+0000","last_clean":"2023-09-30T01:06:31.664042+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:06:31.664042+0000","last_undegraded":"2023-09-30T01:06:31.664042+0000","last_fullsized":"2023-09-30T01:06:31.664042+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:16:33.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666733+0000","last_change":"2023-09-30T01:06:31.665194+0000","last_active":"2023-09-30T01:06:31.666733+0000","last_peered":"2023-09-30T01:06:31.666733+0000","last_clean":"2023-09-30T01:06:31.666733+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:06:31.666733+0000","last_undegraded":"2023-09-30T01:06:31.666733+0000","last_fullsized":"2023-09-30T01:06:31.666733+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:57:04.234397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.666542+0000","last_change":"2023-09-30T01:06:31.664895+0000","last_active":"2023-09-30T01:06:31.666542+0000","last_peered":"2023-09-30T01:06:31.666542+0000","last_clean":"2023-09-30T01:06:31.666542+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:06:31.666542+0000","last_undegraded":"2023-09-30T01:06:31.666542+0000","last_fullsized":"2023-09-30T01:06:31.666542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:23.197758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661339+0000","last_change":"2023-09-30T01:06:31.660865+0000","last_active":"2023-09-30T01:06:31.661339+0000","last_peered":"2023-09-30T01:06:31.661339+0000","last_clean":"2023-09-30T01:06:31.661339+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:06:31.661339+0000","last_undegraded":"2023-09-30T01:06:31.661339+0000","last_fullsized":"2023-09-30T01:06:31.661339+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:05.331355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664420+0000","last_change":"2023-09-30T01:06:31.658586+0000","last_active":"2023-09-30T01:06:31.664420+0000","last_peered":"2023-09-30T01:06:31.664420+0000","last_clean":"2023-09-30T01:06:31.664420+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:06:31.664420+0000","last_undegraded":"2023-09-30T01:06:31.664420+0000","last_fullsized":"2023-09-30T01:06:31.664420+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:14:18.689341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:32.234809+0000","last_change":"2023-09-30T01:06:31.663585+0000","last_active":"2023-09-30T01:06:32.234809+0000","last_peered":"2023-09-30T01:06:32.234809+0000","last_clean":"2023-09-30T01:06:32.234809+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:06:32.234809+0000","last_undegraded":"2023-09-30T01:06:32.234809+0000","last_fullsized":"2023-09-30T01:06:32.234809+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:28:00.773713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.664060+0000","last_change":"2023-09-30T01:06:31.662676+0000","last_active":"2023-09-30T01:06:31.664060+0000","last_peered":"2023-09-30T01:06:31.664060+0000","last_clean":"2023-09-30T01:06:31.664060+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:06:31.664060+0000","last_undegraded":"2023-09-30T01:06:31.664060+0000","last_fullsized":"2023-09-30T01:06:31.664060+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:23:44.116198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.661334+0000","last_change":"2023-09-30T01:06:31.660666+0000","last_active":"2023-09-30T01:06:31.661334+0000","last_peered":"2023-09-30T01:06:31.661334+0000","last_clean":"2023-09-30T01:06:31.661334+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:06:31.661334+0000","last_undegraded":"2023-09-30T01:06:31.661334+0000","last_fullsized":"2023-09-30T01:06:31.661334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:58:38.509668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-09-30T01:06:31.669292+0000","last_change":"2023-09-30T01:06:31.665693+0000","last_active":"2023-09-30T01:06:31.669292+0000","last_peered":"2023-09-30T01:06:31.669292+0000","last_clean":"2023-09-30T01:06:31.669292+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:06:31.669292+0000","last_undegraded":"2023-09-30T01:06:31.669292+0000","last_fullsized":"2023-09-30T01:06:31.669292+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:06:30.641221+0000","last_clean_scrub_stamp":"2023-09-30T01:06:30.641221+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:10.320073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738432,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.216,"5min":0.189,"15min":0.189},"max":{"1min":0.42999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.33100000000000002,"15min":0.33300000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.41499999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.34000000000000002,"15min":0.35099999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.436,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.255},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.22900000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.46200000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366744,"statfs":{"total":96636764160,"available":96631545856,"internally_reserved":0,"allocated":98304,"data_stored":24192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29399999999999998},"min":{"1min":0.23499999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.42899999999999999,"5min":0.498,"15min":0.498},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35099999999999998,"15min":0.33200000000000002},"min":{"1min":0.25900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.48199999999999998,"5min":0.503,"15min":0.503},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:10:36 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.433,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.30599999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.312,"15min":0.31900000000000001},"min":{"1min":0.189,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40400000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30599999999999999,"15min":0.30299999999999999},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46899999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33200000000000002,"15min":0.33400000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.44600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.373},{"interface":"front","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.32200000000000001},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.45400000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.740 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.741 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.741 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2023-09-30T01:06:30.641221+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=6, tm_sec=30, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=11, tm_sec=38, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T01:11:41.741 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T01:12:01.742 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:12:02.063 INFO:teuthology.orchestra.run.smithi079.stdout: 2023-09-30T01:12:02.065 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2023-09-30T01:12:02.076 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2023-09-30T01:12:00.624369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16088,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":15680,"kb_avail":283099432,"statfs":{"total":289910292480,"available":289893818368,"internally_reserved":0,"allocated":393216,"data_stored":1006488,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":16056320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:50.019483+0000","last_change":"2023-09-30T01:11:50.019454+0000","last_active":"2023-09-30T01:11:50.019483+0000","last_peered":"2023-09-30T01:11:50.019483+0000","last_clean":"2023-09-30T01:11:50.019483+0000","last_became_active":"2023-09-30T01:06:31.662359+0000","last_became_peered":"2023-09-30T01:06:31.662359+0000","last_unstale":"2023-09-30T01:11:50.019483+0000","last_undegraded":"2023-09-30T01:11:50.019483+0000","last_fullsized":"2023-09-30T01:11:50.019483+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:50.019339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:50.019339+0000","last_clean_scrub_stamp":"2023-09-30T01:11:50.019339+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:53:44.972721+0000","scrub_duration":0.0014671389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:49.000179+0000","last_change":"2023-09-30T01:11:49.000153+0000","last_active":"2023-09-30T01:11:49.000179+0000","last_peered":"2023-09-30T01:11:49.000179+0000","last_clean":"2023-09-30T01:11:49.000179+0000","last_became_active":"2023-09-30T01:06:31.664904+0000","last_became_peered":"2023-09-30T01:06:31.664904+0000","last_unstale":"2023-09-30T01:11:49.000179+0000","last_undegraded":"2023-09-30T01:11:49.000179+0000","last_fullsized":"2023-09-30T01:11:49.000179+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:49.000051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:49.000051+0000","last_clean_scrub_stamp":"2023-09-30T01:11:49.000051+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:26:21.635699+0000","scrub_duration":0.0013677019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:50.635284+0000","last_change":"2023-09-30T01:11:50.635247+0000","last_active":"2023-09-30T01:11:50.635284+0000","last_peered":"2023-09-30T01:11:50.635284+0000","last_clean":"2023-09-30T01:11:50.635284+0000","last_became_active":"2023-09-30T01:06:31.664647+0000","last_became_peered":"2023-09-30T01:06:31.664647+0000","last_unstale":"2023-09-30T01:11:50.635284+0000","last_undegraded":"2023-09-30T01:11:50.635284+0000","last_fullsized":"2023-09-30T01:11:50.635284+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:50.635145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:50.635145+0000","last_clean_scrub_stamp":"2023-09-30T01:11:50.635145+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:01:27.827231+0000","scrub_duration":0.001480409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:52.086973+0000","last_change":"2023-09-30T01:11:52.086948+0000","last_active":"2023-09-30T01:11:52.086973+0000","last_peered":"2023-09-30T01:11:52.086973+0000","last_clean":"2023-09-30T01:11:52.086973+0000","last_became_active":"2023-09-30T01:06:31.660357+0000","last_became_peered":"2023-09-30T01:06:31.660357+0000","last_unstale":"2023-09-30T01:11:52.086973+0000","last_undegraded":"2023-09-30T01:11:52.086973+0000","last_fullsized":"2023-09-30T01:11:52.086973+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:52.086854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:52.086854+0000","last_clean_scrub_stamp":"2023-09-30T01:11:52.086854+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:31:21.033775+0000","scrub_duration":0.0013270059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:40.973509+0000","last_change":"2023-09-30T01:11:40.973489+0000","last_active":"2023-09-30T01:11:40.973509+0000","last_peered":"2023-09-30T01:11:40.973509+0000","last_clean":"2023-09-30T01:11:40.973509+0000","last_became_active":"2023-09-30T01:06:31.658297+0000","last_became_peered":"2023-09-30T01:06:31.658297+0000","last_unstale":"2023-09-30T01:11:40.973509+0000","last_undegraded":"2023-09-30T01:11:40.973509+0000","last_fullsized":"2023-09-30T01:11:40.973509+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:40.973400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:40.973400+0000","last_clean_scrub_stamp":"2023-09-30T01:11:40.973400+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:12:58.842001+0000","scrub_duration":0.0013486279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'99","reported_seq":149,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:48.639183+0000","last_change":"2023-09-30T01:11:48.639154+0000","last_active":"2023-09-30T01:11:48.639183+0000","last_peered":"2023-09-30T01:11:48.639183+0000","last_clean":"2023-09-30T01:11:48.639183+0000","last_became_active":"2023-09-30T01:06:31.662542+0000","last_became_peered":"2023-09-30T01:06:31.662542+0000","last_unstale":"2023-09-30T01:11:48.639183+0000","last_undegraded":"2023-09-30T01:11:48.639183+0000","last_fullsized":"2023-09-30T01:11:48.639183+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'99","last_scrub_stamp":"2023-09-30T01:11:48.639048+0000","last_deep_scrub":"11'99","last_deep_scrub_stamp":"2023-09-30T01:11:48.639048+0000","last_clean_scrub_stamp":"2023-09-30T01:11:48.639048+0000","objects_scrubbed":2,"log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:04:12.985806+0000","scrub_duration":0.0043107839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:44.026268+0000","last_change":"2023-09-30T01:11:44.026229+0000","last_active":"2023-09-30T01:11:44.026268+0000","last_peered":"2023-09-30T01:11:44.026268+0000","last_clean":"2023-09-30T01:11:44.026268+0000","last_became_active":"2023-09-30T01:06:31.662340+0000","last_became_peered":"2023-09-30T01:06:31.662340+0000","last_unstale":"2023-09-30T01:11:44.026268+0000","last_undegraded":"2023-09-30T01:11:44.026268+0000","last_fullsized":"2023-09-30T01:11:44.026268+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:44.026126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:44.026126+0000","last_clean_scrub_stamp":"2023-09-30T01:11:44.026126+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:30:32.306111+0000","scrub_duration":0.001404258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:51.069624+0000","last_change":"2023-09-30T01:11:51.069600+0000","last_active":"2023-09-30T01:11:51.069624+0000","last_peered":"2023-09-30T01:11:51.069624+0000","last_clean":"2023-09-30T01:11:51.069624+0000","last_became_active":"2023-09-30T01:06:31.660367+0000","last_became_peered":"2023-09-30T01:06:31.660367+0000","last_unstale":"2023-09-30T01:11:51.069624+0000","last_undegraded":"2023-09-30T01:11:51.069624+0000","last_fullsized":"2023-09-30T01:11:51.069624+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:51.069497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:51.069497+0000","last_clean_scrub_stamp":"2023-09-30T01:11:51.069497+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:31:44.135313+0000","scrub_duration":0.0013440749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-30T01:11:51.664144+0000","last_change":"2023-09-30T01:11:51.664119+0000","last_active":"2023-09-30T01:11:51.664144+0000","last_peered":"2023-09-30T01:11:51.664144+0000","last_clean":"2023-09-30T01:11:51.664144+0000","last_became_active":"2023-09-30T01:06:31.665413+0000","last_became_peered":"2023-09-30T01:06:31.665413+0000","last_unstale":"2023-09-30T01:11:51.664144+0000","last_undegraded":"2023-09-30T01:11:51.664144+0000","last_fullsized":"2023-09-30T01:11:51.664144+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:11:51.664014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:11:51.664014+0000","last_clean_scrub_stamp":"2023-09-30T01:11:51.664014+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:44:34.210203+0000","scrub_duration":0.00152586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":933912,"data_compressed":10416,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738436,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5464,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94366376,"statfs":{"total":96636764160,"available":96631169024,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:11:38 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.32900000000000001},"min":{"1min":0.26100000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.436,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.47999999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.35599999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:11:38 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.35099999999999998},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.46400000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.35399999999999998,"15min":0.35499999999999998},"min":{"1min":0.22700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5096,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366744,"statfs":{"total":96636764160,"available":96631545856,"internally_reserved":0,"allocated":98304,"data_stored":24192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:11:38 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.32600000000000001,"15min":0.29599999999999999},"min":{"1min":0.20399999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.41399999999999998,"5min":0.498,"15min":0.498},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35799999999999998,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.215,"15min":0.215},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:11:38 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32200000000000001,"15min":0.307},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45900000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.309},"min":{"1min":0.23100000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.35199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738437,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5528,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":94366312,"statfs":{"total":96636764160,"available":96631103488,"internally_reserved":0,"allocated":147456,"data_stored":491148,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:11:41 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.308,"15min":0.318},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.41199999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.313,"15min":0.30499999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:11:41 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.32800000000000001,"15min":0.33200000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.45200000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.308},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.32900000000000001,"15min":0.32300000000000001},"min":{"1min":0.23200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.44900000000000001,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":466956,"data_compressed":5208,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:12:02.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T01:12:02.405 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:12:02.409 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T01:12:02.409 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T01:12:02.410 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T01:12:02.410 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.442 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T01:12:02.442 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T01:12:02.442 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.506 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T01:12:02.506 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T01:12:02.506 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.576 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T01:12:02.576 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T01:12:02.576 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T01:12:02.576 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.639 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T01:12:02.639 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T01:12:02.639 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T01:12:02.639 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.689 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T01:12:02.689 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T01:12:02.689 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.738 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T01:12:02.739 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T01:12:02.739 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T01:12:02.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-30T01:12:02.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-30T01:12:02.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-09-30T01:12:02.789 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T01:12:02.789 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T01:12:02.789 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-09-30T01:12:02.862 INFO:teuthology.orchestra.run.smithi079.stdout:1696035990.6384542 mon.a (mon.0) 55 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:12:02.863 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T01:12:02.863 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-09-30T01:12:02.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'had wrong client addr' | egrep -v 'had wrong cluster addr' | egrep -v 'must scrub before tier agent can activate' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping updates' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PG_NUM\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(POOL_BACKFILLFULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(POOL_NEARFULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(AUTH_BAD_CAPS\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-09-30T01:12:03.010 INFO:teuthology.orchestra.run.smithi079.stdout:1696035990.6384542 mon.a (mon.0) 55 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:12:03.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:12:03.011 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T01:12:03.200 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:12:03.200 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T01:12:03.281 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:12:03.281 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T01:12:03.394 INFO:tasks.ceph:Archiving mon data... 2023-09-30T01:12:03.395 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/data/mon.a.tgz 2023-09-30T01:12:03.395 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2023-09-30T01:12:03.413 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.7iZ2SegYhi 2023-09-30T01:12:03.413 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7iZ2SegYhi 2023-09-30T01:12:03.521 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.7iZ2SegYhi 2023-09-30T01:12:03.608 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.7iZ2SegYhi is 189KB 2023-09-30T01:12:03.661 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.7iZ2SegYhi 2023-09-30T01:12:03.679 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/data/mon.b.tgz 2023-09-30T01:12:03.679 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2023-09-30T01:12:03.736 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.6qzDLlCjVn 2023-09-30T01:12:03.736 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6qzDLlCjVn 2023-09-30T01:12:03.850 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.6qzDLlCjVn 2023-09-30T01:12:03.939 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.6qzDLlCjVn is 255KB 2023-09-30T01:12:04.033 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.6qzDLlCjVn 2023-09-30T01:12:04.051 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/data/mon.c.tgz 2023-09-30T01:12:04.051 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2023-09-30T01:12:04.107 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.RitTSuMFk5 2023-09-30T01:12:04.107 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.RitTSuMFk5 2023-09-30T01:12:04.222 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.RitTSuMFk5 2023-09-30T01:12:04.310 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.RitTSuMFk5 is 255KB 2023-09-30T01:12:04.403 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.RitTSuMFk5 2023-09-30T01:12:04.421 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T01:12:04.421 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T01:12:04.487 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T01:12:04.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T01:12:04.556 INFO:teuthology.orchestra.run.smithi079.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T01:12:04.561 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T01:12:04.563 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/remote/smithi079/crash 2023-09-30T01:12:04.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T01:12:04.631 INFO:tasks.ceph:Compressing logs... 2023-09-30T01:12:04.631 DEBUG:teuthology.orchestra.run.smithi079:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:12:05.472 INFO:tasks.ceph:Archiving logs... 2023-09-30T01:12:05.473 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/remote/smithi079/log 2023-09-30T01:12:05.474 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T01:12:05.705 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:12:05.770 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9964c8a6ba554178a129dad1cfd7d1f5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T01:12:05.773 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T01:12:05.792 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T01:12:05.833 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T01:12:05.834 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T01:12:05.924 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T01:12:05.941 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T01:12:05.941 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:12:05.958 INFO:teuthology.orchestra.run.smithi079.stderr:bash: ntpq: command not found 2023-09-30T01:12:05.962 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:12:05.963 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2023-09-30T01:12:05.963 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 -208us[ -121us] +/- 74ms 2023-09-30T01:12:05.963 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -273us[ -186us] +/- 55ms 2023-09-30T01:12:05.963 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +542us[ +629us] +/- 60ms 2023-09-30T01:12:05.963 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:12:05.964 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T01:12:05.973 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T01:12:05.973 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T01:12:06.025 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T01:12:06.115 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:12:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:12:06.148 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T01:12:06.156 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T01:12:06.193 INFO:teuthology.task.internal:Duration was 1033.648791 seconds 2023-09-30T01:12:06.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T01:12:06.243 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T01:12:06.243 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T01:12:06.290 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:12:06.518 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T01:12:06.519 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2023-09-30T01:12:06.519 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T01:12:06.542 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T01:12:06.543 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:12:06.622 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T01:12:06.622 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T01:12:06.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T01:12:06.740 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T01:12:06.740 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T01:12:06.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T01:12:06.784 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:12:06.834 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2023-09-30T01:12:06.850 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:12:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:12:06.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T01:12:06.917 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T01:12:06.918 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857/remote/smithi079 2023-09-30T01:12:06.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T01:12:06.986 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T01:12:06.986 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T01:12:07.038 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T01:12:07.048 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T01:12:07.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T01:12:07.099 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T01:12:07.099 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T01:12:07.117 INFO:teuthology.orchestra.run.smithi079.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 01:12 /home/ubuntu/cephtest 2023-09-30T01:12:07.118 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T01:12:07.137 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T01:12:07.236 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T01:12:07.263 DEBUG:teuthology.nuke:shortname: smithi079 2023-09-30T01:12:07.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:12:07.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:46:55.339175', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYhXcNqehS9ibtgkT9dvrVPoGR12Jzkh5yUUUPDQnQYb91PYMztoOBzxgImY6KdFoj+KioZ6uczc6LRFMBJuGM='} 2023-09-30T01:12:07.331 INFO:teuthology.orchestra.console.smithi079:Power off 2023-09-30T01:12:07.332 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T01:12:07.356 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2023-09-30T01:12:07.356 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:12:07.379 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2023-09-30T01:12:07.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-30T01:12:07.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-30T01:12:07.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-09-30T01:12:11.381 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:12:11.491 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2023-09-30T01:12:15.494 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:12:15.603 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2023-09-30T01:12:19.606 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:12:19.716 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2023-09-30T01:12:19.816 INFO:teuthology.orchestra.console.smithi079:Power off completed 2023-09-30T01:12:19.921 INFO:teuthology.run:Summary data: description: rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} duration: 1033.6487908363342 failure_reason: '"1696035990.6384542 mon.a (mon.0) 55 : cluster [ERR] Health check failed: Module ''pg_autoscaler'' has failed: ''PgAutoscaler'' object has no attribute ''noautoscale'' (MGR_MODULE_ERROR)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9964c8a6ba554178a129dad1cfd7d1f5 status: fail success: false 2023-09-30T01:12:19.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:12:20.008 INFO:teuthology.run:FAIL