2023-09-30T00:10:33.968 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:10:33.969 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704 --verbose --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --description rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/orig.config.yaml 2023-09-30T00:10:34.015 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:10:34.083 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7406704' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - - osd.3 - osd.4 - osd.5 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsY3dkdayRkNgoX0Qlg5cj5pGUZND1jWoYlelzq7aQsmzcxZPmCnUaXm9pGTailhMqhJwBVeBbAoLWLY3KOQhs= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrRO3VWph3/GHDDdUoVzBsEbNzIV+12yJdhkmN9+h88n62+fdyd3eNP3rnljTbH85R5E4rDMHojjQVNTbhUoTk= tasks: - install: null - ceph: log-ignorelist: - but it is still running - slow request - overall HEALTH_ - \(CACHE_POOL_ pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 500 - background_exec: mon.a: - while true - do sleep 30 - sudo ceph osd pool set cache cache_target_full_ratio .001 - echo cache-try-flush-evict-all - rados -p cache cache-try-flush-evict-all - sleep 5 - echo cache-flush-evict-all - rados -p cache cache-flush-evict-all - sleep 5 - echo remove overlay - sudo ceph osd tier remove-overlay base - sleep 20 - done - rados: clients: - client.0 max_seconds: 600 objects: 10000 op_weights: copy_from: 50 delete: 50 read: 100 write: 100 ops: 400000 pools: - base size: 1024 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:10:34.084 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:10:34.085 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:10:34.085 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:10:34.086 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:10:34.109 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:10:34.109 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:10:34.109 INFO:teuthology.packaging:ref: None 2023-09-30T00:10:34.109 INFO:teuthology.packaging:tag: None 2023-09-30T00:10:34.109 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:10:34.109 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:10:34.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:10:34.246 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:10:34.246 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:10:34.252 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:10:34.252 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:10:34.265 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:10:34.278 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:10:34.352 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:10:34.376 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:05:11.697690', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsY3dkdayRkNgoX0Qlg5cj5pGUZND1jWoYlelzq7aQsmzcxZPmCnUaXm9pGTailhMqhJwBVeBbAoLWLY3KOQhs='} 2023-09-30T00:10:34.397 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:05:11.695719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrRO3VWph3/GHDDdUoVzBsEbNzIV+12yJdhkmN9+h88n62+fdyd3eNP3rnljTbH85R5E4rDMHojjQVNTbhUoTk='} 2023-09-30T00:10:34.397 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:10:34.411 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2023-09-30T00:10:34.411 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-09-30T00:10:34.412 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:10:34.478 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f02bb998f70>, signals=[15]) 2023-09-30T00:10:34.479 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:10:34.484 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:10:34.485 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:10:34.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:10:34.558 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:10:34.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:10:34.625 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:10:34.631 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-09-30T00:10:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-09-30T00:10:34.654 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-09-30T00:10:34.715 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-09-30T00:10:34.715 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8" 2023-09-30T00:10:34.715 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-09-30T00:10:34.715 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:10:34.716 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:10:34.717 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-09-30T00:10:34.744 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2023-09-30T00:10:34.767 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2023-09-30T00:10:34.767 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2023-09-30T00:10:34.827 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2023-09-30T00:10:34.827 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8" 2023-09-30T00:10:34.827 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2023-09-30T00:10:34.827 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:10:34.828 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:10:34.829 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2023-09-30T00:10:34.852 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:10:34.860 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:10:34.868 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:10:34.868 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:10:34.870 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:10:34.891 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:10:34.897 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:10:34.897 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:10:34.933 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:10:34.960 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:10:35.018 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:10:35.034 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:10:35.034 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:10:35.034 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:10:35.035 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:10:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:10:35.055 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2023-09-30T00:10:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:10:35.056 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-09-30T00:10:35.116 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:10:35.117 INFO:teuthology.task.kernel:Running kernel on smithi173: 4.18.0-489.el8.x86_64 2023-09-30T00:10:35.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2023-09-30T00:10:35.118 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:10:35.118 INFO:teuthology.task.kernel:Running kernel on smithi029: 4.18.0-489.el8.x86_64 2023-09-30T00:10:35.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2023-09-30T00:19:01.523 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 68 kB/s | 33 MB 08:22 2023-09-30T00:19:15.441 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 6.5 MB/s | 48 MB 00:07 2023-09-30T00:19:22.925 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 90 kB/s | 18 kB 00:00 2023-09-30T00:19:23.520 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:19:23.872 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-09-30T00:19:26.135 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 16 MB 00:02 2023-09-30T00:19:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 487 kB/s | 24 kB 00:00 2023-09-30T00:19:33.943 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:19:33.944 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:19:34.234 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:19:34.235 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:19:34.236 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:19:34.236 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:19:34.236 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:19:34.236 INFO:teuthology.orchestra.run.smithi029.stdout:Install 3 Packages 2023-09-30T00:19:34.236 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:19:34.236 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 88 M 2023-09-30T00:19:34.237 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 96 M 2023-09-30T00:19:34.237 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:19:35.179 INFO:teuthology.orchestra.run.smithi029.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2023-09-30T00:19:35.913 INFO:teuthology.orchestra.run.smithi029.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 24 MB/s | 35 MB 00:01 2023-09-30T00:19:36.672 INFO:teuthology.orchestra.run.smithi029.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2023-09-30T00:19:36.672 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:19:36.672 INFO:teuthology.orchestra.run.smithi029.stdout:Total 36 MB/s | 88 MB 00:02 2023-09-30T00:19:37.376 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:19:37.869 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:19:37.869 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:19:39.145 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:19:39.177 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:19:42.104 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:19:42.722 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:19:46.455 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:19:47.005 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:19:50.633 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:19:51.032 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:20:15.693 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:20:15.717 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:20:16.094 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:20:16.094 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:20:16.094 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:20:16.889 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:20:17.074 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:20:17.297 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:20:17.297 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:20:17.298 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:20:17.298 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:20:17.298 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:20:17.298 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:20:17.298 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:20:17.298 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:20:17.519 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:20:17.520 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:20:17.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-09-30T00:20:17.555 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-09-30T00:20:17.555 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-09-30T00:20:17.555 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.02 2023-09-30T00:20:17.555 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 129.el8 2023-09-30T00:20:17.555 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Group : System Environment/Base 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 9265586 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:20:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:Relocations : (not relocatable) 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : CentOS BuildSystem 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-09-30T00:20:17.557 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:20:17.558 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:20:17.558 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:20:17.559 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:20:17.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:20:19.506 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-09-30T00:20:21.650 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-09-30T00:20:21.656 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-09-30T00:20:21.749 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.iguDbn88Go 2023-09-30T00:20:21.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iguDbn88Go 2023-09-30T00:20:21.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.iguDbn88Go 2023-09-30T00:20:21.934 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.iguDbn88Go is 6KB 2023-09-30T00:20:21.982 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.iguDbn88Go 2023-09-30T00:20:21.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:20:22.074 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:20:22.075 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:20:22.075 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:20:22.075 INFO:teuthology.orchestra.run.smithi029.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:20:22.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:20:22.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-09-30T00:20:52.252 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:20:52.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-09-30T00:20:52.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:21:10.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2023-09-30T00:21:19.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-09-30T00:21:19.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:22:19.654 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:22:31.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-09-30T00:22:31.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:22:31.972 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-09-30T00:22:32.340 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-09-30T00:22:32.340 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:22:32.340 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-09-30T00:22:32.362 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:22:32.363 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:22:32.363 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:22:32.363 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:22:33.363 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:22:33.364 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:22:33.459 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2023-09-30T00:22:33.476 DEBUG:teuthology.parallel:result is None 2023-09-30T00:23:13.421 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 45 kB/s | 33 MB 12:35 2023-09-30T00:23:24.904 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 8.7 MB/s | 48 MB 00:05 2023-09-30T00:23:31.987 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 90 kB/s | 18 kB 00:00 2023-09-30T00:23:32.571 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:23:32.856 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-09-30T00:23:33.948 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-09-30T00:23:37.782 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 420 kB/s | 24 kB 00:00 2023-09-30T00:23:41.381 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:23:41.381 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:23:41.635 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:23:41.636 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:23:41.637 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:23:41.637 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:23:41.637 INFO:teuthology.orchestra.run.smithi173.stdout:Install 3 Packages 2023-09-30T00:23:41.637 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:23:41.637 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 88 M 2023-09-30T00:23:41.638 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 96 M 2023-09-30T00:23:41.638 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:23:42.491 INFO:teuthology.orchestra.run.smithi173.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2023-09-30T00:23:43.333 INFO:teuthology.orchestra.run.smithi173.stdout:(2/3): kernel-core-4.18.0-514.el8.x86_64.rpm 28 MB/s | 43 MB 00:01 2023-09-30T00:23:43.700 INFO:teuthology.orchestra.run.smithi173.stdout:(3/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 19 MB/s | 35 MB 00:01 2023-09-30T00:23:43.701 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:23:43.701 INFO:teuthology.orchestra.run.smithi173.stdout:Total 43 MB/s | 88 MB 00:02 2023-09-30T00:23:44.367 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:23:44.782 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:23:44.782 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:23:45.993 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:23:46.010 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:23:48.759 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:23:49.352 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:23:52.903 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:23:53.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:23:56.714 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:23:57.048 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:24:18.954 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:24:18.979 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:24:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:24:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:24:19.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:24:20.046 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:24:20.201 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:24:20.421 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:24:20.422 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:24:20.422 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:24:20.422 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:24:20.422 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:24:20.422 INFO:teuthology.task.kernel:Installing distro kernel on osd.3... 2023-09-30T00:24:20.422 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:24:20.423 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:24:20.632 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:24:20.632 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:24:20.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2023-09-30T00:24:20.665 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.02 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 129.el8 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Group : System Environment/Base 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 9265586 2023-09-30T00:24:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Relocations : (not relocatable) 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : CentOS BuildSystem 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2023-09-30T00:24:20.667 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:24:20.668 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:24:20.669 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:24:20.669 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:24:20.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:24:22.412 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2023-09-30T00:24:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:done 2023-09-30T00:24:24.091 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2023-09-30T00:24:24.114 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.2IuS1MFt68 2023-09-30T00:24:24.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2IuS1MFt68 2023-09-30T00:24:24.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.2IuS1MFt68 2023-09-30T00:24:24.380 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.2IuS1MFt68 is 6KB 2023-09-30T00:24:24.427 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.2IuS1MFt68 2023-09-30T00:24:24.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:24:24.511 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:24:24.511 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:24:24.511 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:24:24.512 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:24:24.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:24:24.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2023-09-30T00:24:54.680 INFO:teuthology.task.kernel:Checking client osd.3 for new kernel version... 2023-09-30T00:24:54.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-09-30T00:24:54.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:25:13.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2023-09-30T00:25:22.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-09-30T00:25:22.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:25:25.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2023-09-30T00:25:37.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-09-30T00:25:37.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:26:09.011 DEBUG:teuthology.orchestra.run.smithi173:> true 2023-09-30T00:26:09.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-09-30T00:26:09.273 INFO:teuthology.task.kernel:Checking kernel version of osd.3, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:26:09.273 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2023-09-30T00:26:09.290 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:26:09.291 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:26:09.291 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:26:09.291 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:26:10.292 INFO:teuthology.task.kernel:Enabling kdb on osd.3... 2023-09-30T00:26:10.293 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:26:10.356 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2023-09-30T00:26:10.379 DEBUG:teuthology.parallel:result is None 2023-09-30T00:26:10.379 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:26:10.388 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:26:10.388 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:26:10.391 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:26:10.413 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:26:10.468 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:26:10.499 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:26:10.500 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:26:10.502 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:26:10.544 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:26:10.595 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:26:10.595 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:26:10.598 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:26:10.636 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:26:10.647 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:26:10.669 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:26:10.682 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:26:10.684 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:26:10.691 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:26:10.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:26:10.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:26:10.726 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:26:10.734 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:26:10.735 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:26:10.769 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:26:10.789 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:26:10.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:26:10.936 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:26:11.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:26:11.077 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:26:11.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:26:11.152 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:26:11.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:26:11.262 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:26:11.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:26:11.354 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:26:11.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:26:11.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-09-30T00:26:11.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2023-09-30T00:26:11.523 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:26:11.556 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:26:11.751 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:26:11.760 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:26:11.760 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:26:11.783 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:26:11.792 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:26:11.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-09-30T00:26:11.890 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-09-30T00:26:11.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2023-09-30T00:26:11.994 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2023-09-30T00:26:11.996 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:26:11.996 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-09-30T00:26:12.030 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-09-30T00:26:12.031 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2023-09-30T00:26:12.066 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2023-09-30T00:26:12.066 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:26:12.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:26:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:26:12.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:26:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:26:12.136 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:26:12.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:26:12.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:26:12.219 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:26:12.229 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:26:12.229 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:26:12.251 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:26:12.252 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:34:31.742 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2023-09-30T00:34:31.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-09-30T00:34:31.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:34:31.820 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-09-30T00:34:31.898 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-09-30T00:34:31.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-09-30T00:34:31.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:34:31.966 DEBUG:teuthology.orchestra.run.smithi173:> true 2023-09-30T00:34:32.038 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-09-30T00:34:32.038 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:34:32.049 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:34:32.049 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:34:32.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:34:32.052 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:34:32.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:34:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:34:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:34:32.114 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:34:32.129 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:34:32.182 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-09-30T00:34:32.196 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-09-30T00:34:32.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2023-09-30T00:34:32.212 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:34:32.214 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2023-09-30T00:34:32.229 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:34:32.229 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:34:32.243 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:34:32.296 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2023-09-30T00:34:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:34:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-09-30T00:34:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.300 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.301 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.301 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.307 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2023-09-30T00:34:32.311 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:34:32.311 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2023-09-30T00:34:32.311 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.312 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.312 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.312 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:34:32.313 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:34:32.321 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:34:32.321 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:34:32.321 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:34:32.321 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:34:32.327 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:34:32.327 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:34:32.328 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:34:32.328 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:34:32.330 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:34:32.330 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:34:32.478 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:34:32.478 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:34:32.549 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:34:32.549 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:34:32.791 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:34:32.791 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:34:32.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:34:32.822 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:34:32.822 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:34:32.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:34:32.908 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:34:32.908 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:34:32.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:34:32.945 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:34:32.945 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:34:32.970 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:34:33.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:34:33.038 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2023-09-30T00:34:33.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2023-09-30T00:34:33.110 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:34:33.143 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-09-30T00:34:33.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-09-30T00:34:33.316 INFO:teuthology.orchestra.run.smithi173.stdout:76 files removed 2023-09-30T00:34:33.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2023-09-30T00:34:33.451 INFO:teuthology.orchestra.run.smithi029.stdout:76 files removed 2023-09-30T00:34:33.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2023-09-30T00:34:34.448 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 113 kB/s | 78 kB 00:00 2023-09-30T00:34:34.809 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 91 kB/s | 78 kB 00:00 2023-09-30T00:34:35.015 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 17 kB/s | 9.2 kB 00:00 2023-09-30T00:34:35.304 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 20 kB/s | 9.2 kB 00:00 2023-09-30T00:34:35.450 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 3.6 kB/s | 1.5 kB 00:00 2023-09-30T00:34:35.899 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2023-09-30T00:34:36.650 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 28 MB/s | 33 MB 00:01 2023-09-30T00:34:38.575 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - AppStream 13 MB/s | 33 MB 00:02 2023-09-30T00:34:44.217 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 33 MB/s | 48 MB 00:01 2023-09-30T00:34:47.235 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 48 MB 00:01 2023-09-30T00:34:51.309 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 97 kB/s | 18 kB 00:00 2023-09-30T00:34:51.893 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:34:54.061 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 6.5 MB 00:02 2023-09-30T00:34:55.264 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-09-30T00:34:55.479 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-09-30T00:34:55.874 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:34:58.068 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 6.5 MB 00:02 2023-09-30T00:34:59.629 INFO:teuthology.orchestra.run.smithi029.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-09-30T00:35:00.757 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-09-30T00:35:01.613 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 2.7 MB/s | 16 MB 00:05 2023-09-30T00:35:05.042 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 436 kB/s | 24 kB 00:00 2023-09-30T00:35:05.479 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 495 kB/s | 24 kB 00:00 2023-09-30T00:35:09.921 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:35:09.924 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:35:09.924 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:35:09.924 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:35:09.924 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:35:09.924 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:35:09.925 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:35:09.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout:Install 21 Packages 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2023-09-30T00:35:09.927 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:09.928 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 63 M 2023-09-30T00:35:09.928 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:35:10.062 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:35:10.065 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:35:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:35:10.065 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:35:10.065 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:35:10.065 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:35:10.065 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:35:10.066 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:35:10.067 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout:Install 21 Packages 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2023-09-30T00:35:10.068 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:35:10.069 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 63 M 2023-09-30T00:35:10.069 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:35:12.086 INFO:teuthology.orchestra.run.smithi029.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 3.5 MB/s | 5.2 MB 00:01 2023-09-30T00:35:12.188 INFO:teuthology.orchestra.run.smithi029.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 240 kB/s | 24 kB 00:00 2023-09-30T00:35:12.781 INFO:teuthology.orchestra.run.smithi029.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 1.3 MB/s | 796 kB 00:00 2023-09-30T00:35:13.390 INFO:teuthology.orchestra.run.smithi029.stdout:(4/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 8.5 MB/s | 23 MB 00:02 2023-09-30T00:35:13.423 INFO:teuthology.orchestra.run.smithi029.stdout:(5/23): libradosstriper1-17.2.6-1165.g93dce5e1. 788 kB/s | 506 kB 00:00 2023-09-30T00:35:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 486 kB/s | 45 kB 00:00 2023-09-30T00:35:13.808 INFO:teuthology.orchestra.run.smithi029.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 395 kB/s | 115 kB 00:00 2023-09-30T00:35:14.155 INFO:teuthology.orchestra.run.smithi029.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 548 kB/s | 190 kB 00:00 2023-09-30T00:35:14.301 INFO:teuthology.orchestra.run.smithi029.stdout:(9/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 6.1 MB/s | 5.6 MB 00:00 2023-09-30T00:35:14.393 INFO:teuthology.orchestra.run.smithi029.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 3.8 MB/s | 358 kB 00:00 2023-09-30T00:35:14.477 INFO:teuthology.orchestra.run.smithi029.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.2 MB/s | 107 kB 00:00 2023-09-30T00:35:14.628 INFO:teuthology.orchestra.run.smithi029.stdout:(12/23): python3-rados-17.2.6-1165.g93dce5e1.el 785 kB/s | 370 kB 00:00 2023-09-30T00:35:15.038 INFO:teuthology.orchestra.run.smithi173.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.5 MB/s | 5.2 MB 00:03 2023-09-30T00:35:15.040 INFO:teuthology.orchestra.run.smithi029.stdout:(13/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 3.8 MB/s | 17 MB 00:04 2023-09-30T00:35:15.087 INFO:teuthology.orchestra.run.smithi029.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 580 kB/s | 354 kB 00:00 2023-09-30T00:35:15.129 INFO:teuthology.orchestra.run.smithi029.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 518 kB/s | 259 kB 00:00 2023-09-30T00:35:15.130 INFO:teuthology.orchestra.run.smithi173.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 266 kB/s | 24 kB 00:00 2023-09-30T00:35:15.572 INFO:teuthology.orchestra.run.smithi173.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 1.8 MB/s | 796 kB 00:00 2023-09-30T00:35:15.584 INFO:teuthology.orchestra.run.smithi029.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 81 kB/s | 44 kB 00:00 2023-09-30T00:35:15.616 INFO:teuthology.orchestra.run.smithi029.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 89 kB/s | 47 kB 00:00 2023-09-30T00:35:15.725 INFO:teuthology.orchestra.run.smithi029.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 515 kB/s | 306 kB 00:00 2023-09-30T00:35:15.750 INFO:teuthology.orchestra.run.smithi029.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 355 kB/s | 59 kB 00:00 2023-09-30T00:35:15.775 INFO:teuthology.orchestra.run.smithi029.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 486 kB/s | 75 kB 00:00 2023-09-30T00:35:15.860 INFO:teuthology.orchestra.run.smithi029.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 13 MB/s | 1.7 MB 00:00 2023-09-30T00:35:15.956 INFO:teuthology.orchestra.run.smithi173.stdout:(4/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 5.2 MB/s | 23 MB 00:04 2023-09-30T00:35:15.989 INFO:teuthology.orchestra.run.smithi173.stdout:(5/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.2 MB/s | 506 kB 00:00 2023-09-30T00:35:16.148 INFO:teuthology.orchestra.run.smithi173.stdout:(6/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 3.6 MB/s | 17 MB 00:04 2023-09-30T00:35:16.173 INFO:teuthology.orchestra.run.smithi173.stdout:(7/23): python3-ceph-argparse-17.2.6-1165.g93dc 243 kB/s | 45 kB 00:00 2023-09-30T00:35:16.257 INFO:teuthology.orchestra.run.smithi173.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 1.0 MB/s | 115 kB 00:00 2023-09-30T00:35:16.357 INFO:teuthology.orchestra.run.smithi173.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.0 MB/s | 190 kB 00:00 2023-09-30T00:35:16.474 INFO:teuthology.orchestra.run.smithi173.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 1.7 MB/s | 370 kB 00:00 2023-09-30T00:35:16.582 INFO:teuthology.orchestra.run.smithi173.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 986 kB/s | 107 kB 00:00 2023-09-30T00:35:16.716 INFO:teuthology.orchestra.run.smithi173.stdout:(12/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 1.0 MB/s | 358 kB 00:00 2023-09-30T00:35:16.808 INFO:teuthology.orchestra.run.smithi173.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-09-30T00:35:16.884 INFO:teuthology.orchestra.run.smithi173.stdout:(14/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 6.0 MB/s | 5.6 MB 00:00 2023-09-30T00:35:16.909 INFO:teuthology.orchestra.run.smithi173.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 438 kB/s | 44 kB 00:00 2023-09-30T00:35:16.992 INFO:teuthology.orchestra.run.smithi173.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 941 kB/s | 259 kB 00:00 2023-09-30T00:35:17.068 INFO:teuthology.orchestra.run.smithi173.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 255 kB/s | 47 kB 00:00 2023-09-30T00:35:17.193 INFO:teuthology.orchestra.run.smithi173.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 294 kB/s | 59 kB 00:00 2023-09-30T00:35:17.227 INFO:teuthology.orchestra.run.smithi173.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 963 kB/s | 306 kB 00:00 2023-09-30T00:35:17.285 INFO:teuthology.orchestra.run.smithi173.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 345 kB/s | 75 kB 00:00 2023-09-30T00:35:17.393 INFO:teuthology.orchestra.run.smithi029.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 2.2 MB/s | 3.6 MB 00:01 2023-09-30T00:35:17.461 INFO:teuthology.orchestra.run.smithi173.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 6.5 MB/s | 1.7 MB 00:00 2023-09-30T00:35:17.694 INFO:teuthology.orchestra.run.smithi173.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 8.8 MB/s | 3.6 MB 00:00 2023-09-30T00:35:18.118 INFO:teuthology.orchestra.run.smithi029.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 1.5 MB/s | 3.5 MB 00:02 2023-09-30T00:35:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:35:18.124 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.9 MB/s | 63 MB 00:08 2023-09-30T00:35:18.170 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:35:18.221 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:35:18.221 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:35:18.695 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:35:18.699 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:35:18.836 INFO:teuthology.orchestra.run.smithi173.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 2.2 MB/s | 3.5 MB 00:01 2023-09-30T00:35:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:35:18.842 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.1 MB/s | 63 MB 00:08 2023-09-30T00:35:18.885 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:35:18.931 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:35:18.931 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:35:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:35:19.307 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:35:19.942 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:35:20.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:35:20.437 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:35:20.462 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:35:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:35:20.857 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:35:20.894 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:35:21.243 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:35:21.397 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:35:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:35:21.513 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:35:21.552 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:35:21.819 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:35:21.898 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:35:21.999 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:35:22.145 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:35:22.384 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:35:22.592 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:35:22.598 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:35:22.803 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:35:22.808 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:35:23.034 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:35:23.045 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:35:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:35:23.456 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:35:23.530 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:35:23.615 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:35:23.728 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:35:24.171 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:35:24.340 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:35:24.353 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:35:24.405 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:35:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:35:24.600 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:35:24.613 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:35:24.777 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:35:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:35:25.008 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:35:25.068 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:35:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:35:25.355 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:35:25.407 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:35:25.462 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:35:25.603 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:35:25.724 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:35:25.811 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:35:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:35:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:35:26.123 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:35:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:35:26.508 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:35:26.779 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:35:26.987 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:35:27.253 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:35:29.025 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:35:29.215 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:35:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:35:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:35:30.175 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:35:30.301 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:35:30.331 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:35:30.544 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:35:31.202 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:35:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:35:31.814 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:35:32.078 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:35:49.147 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:35:49.147 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2023-09-30T00:35:49.148 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2023-09-30T00:35:49.148 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2023-09-30T00:35:49.148 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /run 2023-09-30T00:35:49.148 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2023-09-30T00:35:49.149 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2023-09-30T00:35:49.149 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2023-09-30T00:35:49.149 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2023-09-30T00:35:49.149 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2023-09-30T00:35:49.150 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:50.824 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:35:50.900 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:35:50.900 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:35:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:35:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:35:51.845 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:35:51.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:35:51.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:35:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:35:51.848 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.950 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:35:52.951 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:52.952 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:35:53.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2023-09-30T00:35:54.060 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:49 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:35:55.643 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:35:55.645 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2023-09-30T00:35:55.646 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:35:55.647 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 53 M 2023-09-30T00:35:55.647 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 233 M 2023-09-30T00:35:55.647 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /run 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-09-30T00:35:55.945 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-09-30T00:35:55.946 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-09-30T00:35:55.946 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /dev 2023-09-30T00:35:55.946 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:35:56.007 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-09-30T00:35:56.066 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2023-09-30T00:35:56.156 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.5 MB/s | 323 kB 00:00 2023-09-30T00:35:56.216 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2023-09-30T00:35:56.358 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 330 kB/s | 175 kB 00:00 2023-09-30T00:35:57.831 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:35:57.920 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:35:57.920 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:35:58.218 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:35:58.219 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:35:59.071 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:35:59.072 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:35:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T00:35:59.074 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T00:35:59.074 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T00:35:59.074 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:35:59.074 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:35:59.074 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.329 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:36:00.330 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:00.331 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:36:00.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2023-09-30T00:36:01.602 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:57 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:36:03.352 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:36:03.353 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:36:03.353 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:03.353 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:36:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-09-30T00:36:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:03.355 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 53 M 2023-09-30T00:36:03.355 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 233 M 2023-09-30T00:36:03.355 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:36:03.601 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 997 kB/s | 202 kB 00:00 2023-09-30T00:36:03.710 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-09-30T00:36:03.777 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-09-30T00:36:03.836 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T00:36:04.145 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 234 kB/s | 175 kB 00:00 2023-09-30T00:36:10.428 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 7.3 MB/s | 52 MB 00:07 2023-09-30T00:36:10.430 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:10.431 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.4 MB/s | 53 MB 00:07 2023-09-30T00:36:10.477 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:36:10.501 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:36:10.501 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:36:10.897 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:36:10.900 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:36:11.419 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:36:11.728 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T00:36:11.944 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T00:36:12.164 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:36:12.240 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:36:12.440 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T00:36:12.664 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:36:13.656 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 2.9 MB/s | 52 MB 00:17 2023-09-30T00:36:13.658 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:13.659 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.9 MB/s | 53 MB 00:18 2023-09-30T00:36:13.703 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:36:13.724 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:36:13.724 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:36:14.043 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:36:14.046 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:36:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:36:14.800 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T00:36:15.019 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T00:36:15.157 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:36:15.219 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:36:15.415 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T00:36:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:36:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:36:18.480 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:36:18.807 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:36:18.807 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:36:18.807 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:36:18.807 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T00:36:18.807 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T00:36:18.807 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T00:36:19.595 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T00:36:19.595 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:19.595 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-7.el8.x86_64 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:19.596 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:36:19.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2023-09-30T00:36:20.557 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:36:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:36:20.791 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:16 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:36:21.111 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:36:21.111 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:36:21.111 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:36:21.111 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T00:36:21.112 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T00:36:21.112 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T00:36:21.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-7.el8.x86_64 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:36:21.830 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:21.831 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:36:22.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2023-09-30T00:36:22.547 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:36:22.551 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T00:36:22.552 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:36:22.553 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout:Install 32 Packages 2023-09-30T00:36:22.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:22.555 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 28 M 2023-09-30T00:36:22.555 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 110 M 2023-09-30T00:36:22.555 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:36:22.950 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:17 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:36:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 25 kB/s | 6.4 kB 00:00 2023-09-30T00:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.6 MB/s | 1.6 MB 00:01 2023-09-30T00:36:24.501 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:36:24.504 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:36:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T00:36:24.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:36:24.507 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout:Install 32 Packages 2023-09-30T00:36:24.508 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:24.509 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 28 M 2023-09-30T00:36:24.509 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 110 M 2023-09-30T00:36:24.509 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:36:24.632 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 1.3 MB/s | 2.3 MB 00:01 2023-09-30T00:36:25.299 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 357 kB/s | 238 kB 00:00 2023-09-30T00:36:25.508 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 753 kB/s | 157 kB 00:00 2023-09-30T00:36:25.524 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 14 kB/s | 6.4 kB 00:00 2023-09-30T00:36:25.567 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 664 kB/s | 39 kB 00:00 2023-09-30T00:36:25.659 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2023-09-30T00:36:25.709 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2023-09-30T00:36:25.802 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 1.4 MB/s | 3.9 MB 00:02 2023-09-30T00:36:26.057 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2023-09-30T00:36:26.632 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.0 MB/s | 1.6 MB 00:01 2023-09-30T00:36:26.660 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 290 kB/s | 249 kB 00:00 2023-09-30T00:36:26.791 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 1.3 MB/s | 2.3 MB 00:01 2023-09-30T00:36:27.101 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 5.1 MB/s | 17 MB 00:03 2023-09-30T00:36:27.138 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 228 kB/s | 109 kB 00:00 2023-09-30T00:36:27.180 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 571 kB/s | 641 kB 00:01 2023-09-30T00:36:27.191 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 595 kB/s | 238 kB 00:00 2023-09-30T00:36:27.382 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 155 kB/s | 44 kB 00:00 2023-09-30T00:36:27.400 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 3.9 MB 00:01 2023-09-30T00:36:27.425 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 673 kB/s | 157 kB 00:00 2023-09-30T00:36:27.492 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2023-09-30T00:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 309 kB/s | 39 kB 00:00 2023-09-30T00:36:27.551 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 971 kB/s | 57 kB 00:00 2023-09-30T00:36:27.666 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 351 kB/s | 185 kB 00:00 2023-09-30T00:36:27.751 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2023-09-30T00:36:27.767 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 300 kB/s | 173 kB 00:00 2023-09-30T00:36:27.818 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2023-09-30T00:36:27.827 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 903 kB/s | 249 kB 00:00 2023-09-30T00:36:27.868 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 793 kB/s | 384 kB 00:00 2023-09-30T00:36:27.893 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 139 kB/s | 18 kB 00:00 2023-09-30T00:36:27.894 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2023-09-30T00:36:27.994 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 276 kB/s | 49 kB 00:00 2023-09-30T00:36:27.994 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 436 kB/s | 44 kB 00:00 2023-09-30T00:36:28.019 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 388 kB/s | 59 kB 00:00 2023-09-30T00:36:28.061 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.0 MB/s | 641 kB 00:00 2023-09-30T00:36:28.086 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.0 MB/s | 185 kB 00:00 2023-09-30T00:36:28.112 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-09-30T00:36:28.137 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 109 kB/s | 16 kB 00:00 2023-09-30T00:36:28.161 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 5.0 MB/s | 384 kB 00:00 2023-09-30T00:36:28.163 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 131 kB/s | 19 kB 00:00 2023-09-30T00:36:28.187 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-09-30T00:36:28.212 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-09-30T00:36:28.237 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-09-30T00:36:28.263 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 971 kB/s | 49 kB 00:00 2023-09-30T00:36:28.271 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 150 kB/s | 24 kB 00:00 2023-09-30T00:36:28.288 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-09-30T00:36:28.297 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 183 kB/s | 29 kB 00:00 2023-09-30T00:36:28.339 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 156 kB/s | 27 kB 00:00 2023-09-30T00:36:28.364 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-09-30T00:36:28.388 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-09-30T00:36:28.414 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2023-09-30T00:36:28.439 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-09-30T00:36:28.464 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 582 kB/s | 29 kB 00:00 2023-09-30T00:36:28.490 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 548 kB/s | 27 kB 00:00 2023-09-30T00:36:28.523 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 4.3 MB/s | 255 kB 00:00 2023-09-30T00:36:28.549 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2023-09-30T00:36:28.564 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 379 kB/s | 85 kB 00:00 2023-09-30T00:36:28.574 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-09-30T00:36:28.591 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 549 kB/s | 23 kB 00:00 2023-09-30T00:36:28.600 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 777 kB/s | 255 kB 00:00 2023-09-30T00:36:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 746 kB/s | 251 kB 00:00 2023-09-30T00:36:28.676 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 207 kB/s | 23 kB 00:00 2023-09-30T00:36:28.680 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:28.680 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.6 MB/s | 28 MB 00:06 2023-09-30T00:36:28.770 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:36:28.811 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:36:28.811 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:36:29.156 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:36:29.159 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:36:29.650 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 5.5 MB/s | 17 MB 00:03 2023-09-30T00:36:29.651 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:29.651 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.5 MB/s | 28 MB 00:05 2023-09-30T00:36:29.739 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:36:29.777 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:36:29.777 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:36:30.086 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:36:30.089 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:36:30.419 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:36:30.713 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T00:36:30.930 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T00:36:31.216 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T00:36:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:36:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T00:36:31.721 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T00:36:31.858 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T00:36:31.937 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T00:36:32.186 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T00:36:32.375 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T00:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T00:36:32.605 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T00:36:32.757 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T00:36:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T00:36:32.985 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T00:36:33.007 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T00:36:33.193 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T00:36:33.226 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T00:36:33.414 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T00:36:33.523 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T00:36:33.656 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T00:36:33.833 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T00:36:33.860 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T00:36:34.081 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T00:36:34.294 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T00:36:34.380 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T00:36:34.566 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T00:36:34.786 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T00:36:34.815 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T00:36:34.995 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T00:36:35.152 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T00:36:35.222 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:36:35.364 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T00:36:35.539 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T00:36:35.680 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T00:36:35.735 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T00:36:35.931 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:36:35.973 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T00:36:36.217 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T00:36:36.335 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T00:36:36.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T00:36:36.595 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T00:36:36.766 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T00:36:36.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T00:36:37.067 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T00:36:37.079 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T00:36:37.385 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T00:36:37.424 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T00:36:37.685 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T00:36:37.685 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T00:36:38.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T00:36:38.047 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T00:36:38.284 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:36:38.408 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T00:36:38.809 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T00:36:39.222 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T00:36:39.526 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:36:39.772 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:36:40.008 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:36:40.402 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:36:40.604 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:36:40.820 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:36:40.995 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:36:41.014 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:36:41.178 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:36:41.183 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:36:41.451 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:36:41.592 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T00:36:41.593 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T00:36:41.594 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T00:36:41.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T00:36:41.596 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T00:36:41.596 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T00:36:41.890 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:36:42.152 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:36:42.397 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:42.588 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T00:36:42.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T00:36:42.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:36:42.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:36:42.592 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:36:42.593 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:42.593 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:36:42.652 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:36:42.677 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:36:42.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2023-09-30T00:36:43.053 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:36:43.535 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:36:43.535 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T00:36:43.535 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T00:36:43.535 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T00:36:43.536 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T00:36:43.537 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T00:36:43.538 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T00:36:43.538 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T00:36:43.538 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T00:36:43.538 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T00:36:43.539 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T00:36:43.650 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:36:44.827 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:36:44.828 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:36:44.829 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:36:44.830 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:36:44.830 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:36:44.830 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:36:44.830 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:36:44.830 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:44.830 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:36:45.021 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:36:45.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2023-09-30T00:36:45.209 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:36:45.210 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:36:45.210 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:36:45.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2023-09-30T00:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:36:46.167 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:36:47.642 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:36:47.778 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:36:47.779 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:47.780 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 80 k 2023-09-30T00:36:47.780 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 363 k 2023-09-30T00:36:47.780 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:36:47.854 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:36:47.856 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:36:47.856 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:36:48.281 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 160 kB/s | 80 kB 00:00 2023-09-30T00:36:48.281 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:48.281 INFO:teuthology.orchestra.run.smithi173.stdout:Total 160 kB/s | 80 kB 00:00 2023-09-30T00:36:48.282 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:36:48.289 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:36:48.290 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:36:48.291 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:36:48.294 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:36:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:36:48.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2023-09-30T00:36:48.876 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:49.273 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:49.325 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:36:49.928 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:49.928 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:49.928 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:36:49.928 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:36:49.929 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:49.929 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:36:50.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:36:51.031 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:36:51.080 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:36:51.081 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:51.082 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:36:51.082 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:51.082 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 80 k 2023-09-30T00:36:51.082 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 363 k 2023-09-30T00:36:51.082 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:36:51.814 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 109 kB/s | 80 kB 00:00 2023-09-30T00:36:51.815 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:51.815 INFO:teuthology.orchestra.run.smithi029.stdout:Total 109 kB/s | 80 kB 00:00 2023-09-30T00:36:51.816 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:36:51.826 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:36:51.826 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:36:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:36:51.832 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:36:51.895 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:36:52.526 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:52.625 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:36:52.626 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2023-09-30T00:36:52.626 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:52.626 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:36:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:36:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:52.627 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:36:52.627 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:52.627 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:36:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:52.628 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 161 k 2023-09-30T00:36:52.628 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 490 k 2023-09-30T00:36:52.628 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:36:52.746 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:53.007 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:53.020 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 411 kB/s | 161 kB 00:00 2023-09-30T00:36:53.021 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:53.021 INFO:teuthology.orchestra.run.smithi173.stdout:Total 410 kB/s | 161 kB 00:00 2023-09-30T00:36:53.021 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:36:53.032 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:36:53.032 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:36:53.079 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:36:53.082 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:36:53.196 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:36:53.418 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:36:53.691 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:36:53.691 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:53.692 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:36:53.692 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:36:53.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:53.692 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:36:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:36:53.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:36:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:36:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:54.507 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:36:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:54.507 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:36:54.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2023-09-30T00:36:54.878 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:36:55.562 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:36:56.661 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:36:56.661 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:36:56.662 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:56.663 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 161 k 2023-09-30T00:36:56.663 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 490 k 2023-09-30T00:36:56.663 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:36:56.940 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:36:56.959 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 546 kB/s | 161 kB 00:00 2023-09-30T00:36:56.959 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:56.959 INFO:teuthology.orchestra.run.smithi029.stdout:Total 543 kB/s | 161 kB 00:00 2023-09-30T00:36:56.960 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:36:56.974 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:36:56.974 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:36:57.033 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:36:57.036 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:36:57.128 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:36:57.129 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:36:57.129 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:36:57.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:36:57.400 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:36:57.660 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:36:58.080 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:53 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:36:58.137 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:36:58.804 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:36:58.804 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:58.804 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:36:58.804 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:58.804 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:36:58.804 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:36:59.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2023-09-30T00:36:59.671 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:36:59.672 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:36:59.673 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:59.674 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:36:59.674 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:36:59.674 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2023-09-30T00:36:59.674 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:36:59.674 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2023-09-30T00:36:59.674 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 8.0 M 2023-09-30T00:36:59.675 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:36:59.983 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:55 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:00.041 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 94 kB/s | 14 kB 00:00 2023-09-30T00:37:00.133 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-grafana-dashboards-17.2.6-1165.g93d 94 kB/s | 23 kB 00:00 2023-09-30T00:37:00.191 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 284 kB/s | 43 kB 00:00 2023-09-30T00:37:00.410 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 3.2 MB/s | 1.7 MB 00:00 2023-09-30T00:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 103 kB/s | 34 kB 00:00 2023-09-30T00:37:00.717 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 374 kB/s | 196 kB 00:00 2023-09-30T00:37:00.719 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:00.719 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2023-09-30T00:37:00.732 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:00.742 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:00.742 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:00.836 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:00.839 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:01.090 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:01.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T00:37:01.529 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:37:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T00:37:01.741 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:01.742 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:37:01.742 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:01.744 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T00:37:01.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:37:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:37:02.257 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:37:02.463 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:37:02.752 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:37:02.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:37:02.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:37:02.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:37:02.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T00:37:02.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T00:37:02.859 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:58 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T00:37:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:03.453 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:03.453 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:37:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:37:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:37:03.454 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:03.454 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:03.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:37:04.440 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:59 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:04.615 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:04.616 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:04.616 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:37:04.616 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:04.616 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:37:04.617 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:37:04.618 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:04.618 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:37:04.618 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:04.618 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2023-09-30T00:37:04.618 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2023-09-30T00:37:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 8.0 M 2023-09-30T00:37:04.619 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:37:05.217 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 102 kB/s | 23 kB 00:00 2023-09-30T00:37:05.243 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 58 kB/s | 14 kB 00:00 2023-09-30T00:37:05.377 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2023-09-30T00:37:05.402 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 231 kB/s | 43 kB 00:00 2023-09-30T00:37:05.471 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 3.5 MB/s | 1.7 MB 00:00 2023-09-30T00:37:05.538 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-09-30T00:37:05.540 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:05.540 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2023-09-30T00:37:05.553 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:37:05.564 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:37:05.564 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:37:05.661 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:37:05.664 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:37:06.002 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:37:06.029 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:06.030 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================================ 2023-09-30T00:37:06.030 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:37:06.030 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================================ 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T00:37:06.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================================ 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout:Install 8 Packages 2023-09-30T00:37:06.032 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:06.033 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 35 M 2023-09-30T00:37:06.033 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 210 M 2023-09-30T00:37:06.033 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T00:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T00:37:06.557 INFO:teuthology.orchestra.run.smithi173.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2023-09-30T00:37:06.624 INFO:teuthology.orchestra.run.smithi173.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.4 MB/s | 4.7 MB 00:00 2023-09-30T00:37:06.708 INFO:teuthology.orchestra.run.smithi173.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2023-09-30T00:37:06.769 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T00:37:06.783 INFO:teuthology.orchestra.run.smithi173.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 17 MB/s | 3.7 MB 00:00 2023-09-30T00:37:06.875 INFO:teuthology.orchestra.run.smithi173.stdout:(5/8): ceph-mgr-diskprediction-local-17.2.6-116 9.8 MB/s | 7.4 MB 00:00 2023-09-30T00:37:06.882 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:37:07.092 INFO:teuthology.orchestra.run.smithi173.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 794 kB/s | 172 kB 00:00 2023-09-30T00:37:07.228 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:37:07.234 INFO:teuthology.orchestra.run.smithi173.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 26 MB/s | 14 MB 00:00 2023-09-30T00:37:07.259 INFO:teuthology.orchestra.run.smithi173.stdout:(8/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2023-09-30T00:37:07.259 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:07.260 INFO:teuthology.orchestra.run.smithi173.stdout:Total 29 MB/s | 35 MB 00:01 2023-09-30T00:37:07.439 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:37:07.493 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:07.511 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:07.512 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:07.821 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:37:07.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:37:07.821 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:37:07.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:37:07.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T00:37:07.822 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T00:37:07.917 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:07.920 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T00:37:08.881 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:37:08.882 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:37:08.883 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:08.883 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:09.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:37:09.141 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:37:09.278 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:37:09.466 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:37:10.015 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:06 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:10.059 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:37:10.324 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:37:10.944 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:37:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:37:11.772 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:11.773 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================================ 2023-09-30T00:37:11.773 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:37:11.773 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================================ 2023-09-30T00:37:11.773 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:37:11.773 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:37:11.774 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================================ 2023-09-30T00:37:11.775 INFO:teuthology.orchestra.run.smithi029.stdout:Install 8 Packages 2023-09-30T00:37:11.775 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:11.775 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 35 M 2023-09-30T00:37:11.775 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 210 M 2023-09-30T00:37:11.775 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:37:11.788 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:37:12.202 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:37:12.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:37:12.662 INFO:teuthology.orchestra.run.smithi029.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 9.8 MB/s | 4.7 MB 00:00 2023-09-30T00:37:12.829 INFO:teuthology.orchestra.run.smithi029.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.5 MB/s | 4.9 MB 00:00 2023-09-30T00:37:12.940 INFO:teuthology.orchestra.run.smithi029.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2023-09-30T00:37:12.965 INFO:teuthology.orchestra.run.smithi029.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2023-09-30T00:37:13.526 INFO:teuthology.orchestra.run.smithi029.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 23 MB/s | 14 MB 00:00 2023-09-30T00:37:13.966 INFO:teuthology.orchestra.run.smithi029.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 645 kB/s | 645 kB 00:01 2023-09-30T00:37:14.270 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:37:14.582 INFO:teuthology.orchestra.run.smithi029.stdout:(7/8): ceph-mgr-diskprediction-local-17.2.6-116 3.1 MB/s | 7.4 MB 00:02 2023-09-30T00:37:15.103 INFO:teuthology.orchestra.run.smithi029.stdout:(8/8): libquadmath-8.5.0-20.el8.x86_64.rpm 109 kB/s | 172 kB 00:01 2023-09-30T00:37:15.104 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:15.104 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 35 MB 00:03 2023-09-30T00:37:15.356 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:37:15.377 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:37:15.377 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:37:15.603 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T00:37:15.841 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:37:15.845 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:37:15.847 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:37:16.160 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:37:16.160 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:37:16.161 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T00:37:16.161 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:37:16.161 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T00:37:16.161 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:37:16.161 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T00:37:16.161 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T00:37:16.448 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:37:16.656 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:37:16.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T00:37:16.834 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:16.835 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:16.863 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:37:16.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:37:17.039 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:37:17.702 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:37:17.818 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:12 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:17.940 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:37:18.693 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:37:18.865 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:37:19.393 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:19.395 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:19.395 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:37:19.395 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:19.395 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:19.395 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:37:19.395 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T00:37:19.396 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:19.397 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 1.9 M 2023-09-30T00:37:19.398 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 25 M 2023-09-30T00:37:19.398 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:19.698 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:37:19.925 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 154 kB/s | 20 kB 00:00 2023-09-30T00:37:19.950 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 172 kB/s | 27 kB 00:00 2023-09-30T00:37:20.051 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-09-30T00:37:20.076 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 877 kB/s | 110 kB 00:00 2023-09-30T00:37:20.139 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:37:20.193 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 119 kB/s | 47 kB 00:00 2023-09-30T00:37:20.227 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 283 kB/s | 43 kB 00:00 2023-09-30T00:37:20.285 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 661 kB/s | 155 kB 00:00 2023-09-30T00:37:20.311 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 281 kB/s | 33 kB 00:00 2023-09-30T00:37:20.336 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 156 kB/s | 16 kB 00:00 2023-09-30T00:37:20.386 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-09-30T00:37:20.412 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 648 kB/s | 81 kB 00:00 2023-09-30T00:37:20.437 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-09-30T00:37:20.439 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:37:20.470 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 7.5 MB/s | 1.2 MB 00:00 2023-09-30T00:37:20.472 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:20.473 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.8 MB/s | 1.9 MB 00:01 2023-09-30T00:37:20.533 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:20.549 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:20.549 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:20.635 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:20.638 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:21.146 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:37:21.614 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T00:37:21.866 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T00:37:22.112 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T00:37:22.294 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T00:37:22.465 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:37:22.568 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T00:37:22.903 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T00:37:23.202 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:37:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T00:37:24.080 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:37:24.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:37:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:37:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:37:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T00:37:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:37:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T00:37:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:37:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T00:37:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T00:37:24.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T00:37:24.794 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:37:25.039 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T00:37:25.213 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:37:25.250 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T00:37:25.250 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:25.250 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:37:25.250 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:25.250 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:25.251 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:25.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:37:25.508 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:37:25.508 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:37:25.508 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T00:37:25.508 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T00:37:25.508 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T00:37:25.508 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T00:37:25.509 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T00:37:26.198 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T00:37:26.198 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:37:26.199 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:37:26.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:37:26.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:37:26.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:37:26.200 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:37:26.200 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:26.200 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:26.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:37:26.391 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:22 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:27.189 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:22 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:28.161 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:37:28.163 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:28.164 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:37:28.165 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:28.165 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2023-09-30T00:37:28.165 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:28.165 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 1.9 M 2023-09-30T00:37:28.165 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 25 M 2023-09-30T00:37:28.165 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:37:28.699 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 173 kB/s | 27 kB 00:00 2023-09-30T00:37:28.724 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 111 kB/s | 20 kB 00:00 2023-09-30T00:37:28.761 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:37:28.762 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout:Install 5 Packages 2023-09-30T00:37:28.763 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:28.764 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.0 M 2023-09-30T00:37:28.764 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2023-09-30T00:37:28.764 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:28.841 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-09-30T00:37:28.867 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 776 kB/s | 110 kB 00:00 2023-09-30T00:37:29.034 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 255 kB/s | 43 kB 00:00 2023-09-30T00:37:29.060 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 92 kB/s | 47 kB 00:00 2023-09-30T00:37:29.085 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 636 kB/s | 155 kB 00:00 2023-09-30T00:37:29.270 INFO:teuthology.orchestra.run.smithi173.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.0 MB/s | 538 kB 00:00 2023-09-30T00:37:29.478 INFO:teuthology.orchestra.run.smithi173.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2023-09-30T00:37:29.502 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 37 kB/s | 16 kB 00:00 2023-09-30T00:37:29.512 INFO:teuthology.orchestra.run.smithi173.stdout:(3/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 239 kB/s | 121 kB 00:00 2023-09-30T00:37:29.528 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 67 kB/s | 33 kB 00:00 2023-09-30T00:37:29.537 INFO:teuthology.orchestra.run.smithi173.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.7 MB/s | 476 kB 00:00 2023-09-30T00:37:29.606 INFO:teuthology.orchestra.run.smithi173.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 474 kB/s | 60 kB 00:00 2023-09-30T00:37:29.606 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:29.606 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.1 MB/s | 6.0 MB 00:00 2023-09-30T00:37:29.645 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 145 kB/s | 81 kB 00:00 2023-09-30T00:37:29.658 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:29.669 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:29.669 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:29.754 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 284 kB/s | 64 kB 00:00 2023-09-30T00:37:29.776 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:29.779 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:29.871 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 272 kB/s | 61 kB 00:00 2023-09-30T00:37:30.118 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:30.188 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2023-09-30T00:37:30.191 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:30.191 INFO:teuthology.orchestra.run.smithi029.stdout:Total 976 kB/s | 1.9 MB 00:02 2023-09-30T00:37:30.256 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:37:30.274 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:37:30.274 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:37:30.375 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:37:30.378 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:37:30.469 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T00:37:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:37:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:37:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:37:31.803 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T00:37:31.858 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T00:37:32.096 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T00:37:32.155 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T00:37:32.330 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T00:37:32.354 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:37:32.647 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T00:37:32.664 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:37:32.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:37:32.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T00:37:32.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T00:37:32.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:37:33.001 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T00:37:33.322 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T00:37:33.431 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:37:33.432 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:33.433 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:33.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2023-09-30T00:37:33.598 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:37:34.405 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:29 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:34.824 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:37:35.188 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T00:37:35.438 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:37:35.750 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T00:37:35.929 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:37:36.007 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:36.008 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:36.008 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:37:36.008 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:36.008 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:36.008 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:37:36.008 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2023-09-30T00:37:36.009 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T00:37:36.009 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:36.009 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:36.009 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:36.009 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2023-09-30T00:37:36.009 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:36.010 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 957 k 2023-09-30T00:37:36.010 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 3.0 M 2023-09-30T00:37:36.010 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:36.237 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 446 kB/s | 83 kB 00:00 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:37:36.297 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T00:37:36.298 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:37:36.298 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T00:37:36.298 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T00:37:36.298 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T00:37:36.879 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 1.0 MB/s | 874 kB 00:00 2023-09-30T00:37:36.879 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:36.880 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.1 MB/s | 957 kB 00:00 2023-09-30T00:37:36.891 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:36.913 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:36.913 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:37.001 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:37.004 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:37.244 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T00:37:37.244 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:37.244 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:37:37.244 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:37:37.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:37:37.246 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:37:37.246 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:37.246 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:37.451 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T00:37:37.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:37:37.711 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:37:38.111 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:37:38.111 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:37:38.445 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:34 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:38.757 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T00:37:38.757 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:38.757 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:38.757 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T00:37:38.757 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:38.757 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:38.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2023-09-30T00:37:39.860 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:34 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:40.212 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:37:40.213 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout:Install 5 Packages 2023-09-30T00:37:40.214 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:40.215 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.0 M 2023-09-30T00:37:40.215 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2023-09-30T00:37:40.215 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:37:40.593 INFO:teuthology.orchestra.run.smithi029.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2023-09-30T00:37:40.845 INFO:teuthology.orchestra.run.smithi029.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 9.1 MB/s | 4.8 MB 00:00 2023-09-30T00:37:40.904 INFO:teuthology.orchestra.run.smithi029.stdout:(3/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 207 kB/s | 121 kB 00:00 2023-09-30T00:37:41.004 INFO:teuthology.orchestra.run.smithi029.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.1 MB/s | 476 kB 00:00 2023-09-30T00:37:41.113 INFO:teuthology.orchestra.run.smithi029.stdout:(5/5): python3-natsort-7.1.1-2.el8.noarch.rpm 225 kB/s | 60 kB 00:00 2023-09-30T00:37:41.115 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:41.116 INFO:teuthology.orchestra.run.smithi029.stdout:Total 6.6 MB/s | 6.0 MB 00:00 2023-09-30T00:37:41.174 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:37:41.187 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:37:41.187 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:37:41.312 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:37:41.316 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:37:41.406 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:41.407 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:41.408 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:37:41.408 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:41.408 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 267 k 2023-09-30T00:37:41.408 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.3 M 2023-09-30T00:37:41.408 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:41.662 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:37:41.974 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 471 kB/s | 267 kB 00:00 2023-09-30T00:37:41.975 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:41.976 INFO:teuthology.orchestra.run.smithi173.stdout:Total 470 kB/s | 267 kB 00:00 2023-09-30T00:37:41.976 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:41.986 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:41.986 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:42.018 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:42.021 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:42.096 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T00:37:42.233 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:42.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:37:42.903 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:37:43.180 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:43.651 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T00:37:43.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2023-09-30T00:37:43.987 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T00:37:44.261 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:37:44.597 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:37:44.597 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:37:44.597 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T00:37:44.598 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T00:37:44.598 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:37:44.631 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:39 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:45.343 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T00:37:45.343 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:45.343 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:37:45.343 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:45.343 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:37:45.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T00:37:45.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T00:37:45.344 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:37:45.344 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:45.344 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:45.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2023-09-30T00:37:46.198 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:46.199 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2023-09-30T00:37:46.199 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:37:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:46.201 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 130 k 2023-09-30T00:37:46.201 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 464 k 2023-09-30T00:37:46.201 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:46.466 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:42 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:47.174 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 133 kB/s | 130 kB 00:00 2023-09-30T00:37:47.174 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:47.175 INFO:teuthology.orchestra.run.smithi173.stdout:Total 133 kB/s | 130 kB 00:00 2023-09-30T00:37:47.175 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:47.236 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:47.239 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:47.389 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:47.676 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:37:47.974 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:37:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T00:37:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 957 k 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 3.0 M 2023-09-30T00:37:48.213 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:37:48.465 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 393 kB/s | 83 kB 00:00 2023-09-30T00:37:48.606 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:37:48.606 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:48.606 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:48.606 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:37:48.606 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:48.606 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:48.757 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 1.7 MB/s | 874 kB 00:00 2023-09-30T00:37:48.758 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:48.758 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.7 MB/s | 957 kB 00:00 2023-09-30T00:37:48.771 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:37:48.795 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:37:48.795 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:37:48.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2023-09-30T00:37:48.895 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:37:48.899 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:37:49.139 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:37:49.451 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T00:37:49.662 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:37:49.730 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:44 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:50.111 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:37:50.112 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:37:50.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T00:37:50.797 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:50.797 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:37:50.797 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T00:37:50.797 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:50.797 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:51.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2023-09-30T00:37:51.119 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:37:51.305 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:51.306 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:37:51.306 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:51.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2023-09-30T00:37:52.051 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:53.827 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:37:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:37:53.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:53.829 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 267 k 2023-09-30T00:37:53.829 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.3 M 2023-09-30T00:37:53.829 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:37:53.871 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:53.871 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:37:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:53.873 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 25 k 2023-09-30T00:37:53.873 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 109 k 2023-09-30T00:37:53.873 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:37:54.310 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 57 kB/s | 25 kB 00:00 2023-09-30T00:37:54.311 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:54.311 INFO:teuthology.orchestra.run.smithi173.stdout:Total 57 kB/s | 25 kB 00:00 2023-09-30T00:37:54.311 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:37:54.315 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:37:54.315 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:37:54.317 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:37:54.320 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:37:54.369 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:37:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:37:54.866 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:37:54.907 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 248 kB/s | 267 kB 00:01 2023-09-30T00:37:54.908 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:37:54.909 INFO:teuthology.orchestra.run.smithi029.stdout:Total 247 kB/s | 267 kB 00:01 2023-09-30T00:37:54.909 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:37:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:37:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:37:55.089 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:37:55.092 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:37:55.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:37:55.451 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:55.451 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:37:55.451 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:37:55.451 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:37:55.451 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:55.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2023-09-30T00:37:55.663 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:37:55.960 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:37:56.427 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:37:56.436 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:37:57.193 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:37:57.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:57.193 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:37:57.193 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:37:57.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:37:57.193 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:37:57.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2023-09-30T00:37:57.831 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:37:58.020 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:37:58.022 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:37:58.022 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:37:58.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2023-09-30T00:37:58.442 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:37:59.019 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:00.204 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:00.204 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:38:00.205 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:00.206 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 130 k 2023-09-30T00:38:00.206 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 464 k 2023-09-30T00:38:00.206 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:38:00.396 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:00.584 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:00.585 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:38:00.585 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:00.586 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 342 kB/s | 130 kB 00:00 2023-09-30T00:38:00.586 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:00.587 INFO:teuthology.orchestra.run.smithi029.stdout:Total 341 kB/s | 130 kB 00:00 2023-09-30T00:38:00.587 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:38:00.594 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:38:00.594 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:38:00.654 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:38:00.658 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:38:00.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2023-09-30T00:38:00.823 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:38:01.168 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:01.499 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:01.571 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:56 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:02.204 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:02.204 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:02.204 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:38:02.204 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:02.205 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:02.205 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:02.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2023-09-30T00:38:02.966 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:03.151 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:03.152 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:38:03.152 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:03.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2023-09-30T00:38:03.415 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:59 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:04.123 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:59 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:04.954 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:05.162 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:05.164 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:05.164 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:05.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2023-09-30T00:38:05.537 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:05.725 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:05.726 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:38:05.727 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:05.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2023-09-30T00:38:06.268 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:02 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:06.725 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:01 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:08.016 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:08.016 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:38:08.017 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:08.018 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 25 k 2023-09-30T00:38:08.018 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 109 k 2023-09-30T00:38:08.018 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:38:08.121 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:08.309 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:08.310 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:38:08.310 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:08.395 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 66 kB/s | 25 kB 00:00 2023-09-30T00:38:08.396 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:08.396 INFO:teuthology.orchestra.run.smithi029.stdout:Total 66 kB/s | 25 kB 00:00 2023-09-30T00:38:08.397 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:38:08.401 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:38:08.401 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:38:08.404 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:38:08.408 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:38:08.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2023-09-30T00:38:08.457 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:38:08.734 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:09.036 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:09.277 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:04 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:09.741 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:38:09.741 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:09.742 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:09.742 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:09.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2023-09-30T00:38:10.671 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:10.859 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:10.860 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2023-09-30T00:38:10.860 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:10.860 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:06 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:10.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2023-09-30T00:38:11.819 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:06 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:12.410 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:12.621 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:12.622 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:12.622 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:12.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2023-09-30T00:38:13.472 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:13.473 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:13.473 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2023-09-30T00:38:13.473 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:13.473 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:38:13.473 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:38:13.474 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:13.474 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:38:13.474 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:13.474 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:38:13.474 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:13.474 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 90 k 2023-09-30T00:38:13.475 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 264 k 2023-09-30T00:38:13.475 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:38:13.722 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:09 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:14.300 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 109 kB/s | 90 kB 00:00 2023-09-30T00:38:14.300 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:14.301 INFO:teuthology.orchestra.run.smithi173.stdout:Total 109 kB/s | 90 kB 00:00 2023-09-30T00:38:14.301 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:38:14.312 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:38:14.312 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:38:14.366 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:38:14.369 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:38:14.478 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:38:14.738 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:15.038 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:15.268 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:15.480 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:15.481 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:15.482 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:15.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2023-09-30T00:38:15.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:15.695 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:15.695 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:38:15.695 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:15.695 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:15.695 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:15.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2023-09-30T00:38:16.586 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:12 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:16.784 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:11 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:18.134 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:18.343 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:18.344 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:18.344 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:18.367 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:18.367 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:18.367 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:38:18.368 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:18.369 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.5 M 2023-09-30T00:38:18.369 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 16 M 2023-09-30T00:38:18.369 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:38:18.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2023-09-30T00:38:19.441 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:15 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:19.802 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 2.4 MB/s | 3.5 MB 00:01 2023-09-30T00:38:19.803 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:19.803 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 3.5 MB 00:01 2023-09-30T00:38:19.804 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:38:19.816 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:38:19.817 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:38:19.884 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:38:19.888 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:38:20.335 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:38:20.580 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:21.007 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:21.023 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:21.230 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:21.232 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:21.232 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:21.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2023-09-30T00:38:21.631 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:21.632 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:21.632 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:38:21.632 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:21.632 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:21.632 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:21.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2023-09-30T00:38:22.319 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:18 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:22.715 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:03:17 ago on Sat 30 Sep 2023 12:35:05 AM UTC. 2023-09-30T00:38:23.870 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:24.081 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:24.082 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:24.083 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:24.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2023-09-30T00:38:24.302 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2023-09-30T00:38:24.302 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2023-09-30T00:38:24.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:24.304 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 173 k 2023-09-30T00:38:24.304 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 552 k 2023-09-30T00:38:24.304 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2023-09-30T00:38:24.960 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 264 kB/s | 173 kB 00:00 2023-09-30T00:38:24.960 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:24.960 INFO:teuthology.orchestra.run.smithi173.stdout:Total 264 kB/s | 173 kB 00:00 2023-09-30T00:38:24.961 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2023-09-30T00:38:24.971 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2023-09-30T00:38:24.972 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2023-09-30T00:38:25.024 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2023-09-30T00:38:25.027 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2023-09-30T00:38:25.151 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2023-09-30T00:38:25.193 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:21 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:25.370 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:25.642 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:26.265 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2023-09-30T00:38:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:26.265 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2023-09-30T00:38:26.465 DEBUG:teuthology.parallel:result is None 2023-09-30T00:38:26.761 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:38:26.971 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:26.972 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2023-09-30T00:38:26.973 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:27.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2023-09-30T00:38:28.077 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:24 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:29.839 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:29.839 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:29.839 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-09-30T00:38:29.839 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:29.839 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:38:29.840 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:38:29.840 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:29.840 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:38:29.840 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:29.840 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:38:29.840 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:29.841 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 90 k 2023-09-30T00:38:29.841 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 264 k 2023-09-30T00:38:29.841 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:38:30.215 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 241 kB/s | 90 kB 00:00 2023-09-30T00:38:30.216 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:30.216 INFO:teuthology.orchestra.run.smithi029.stdout:Total 240 kB/s | 90 kB 00:00 2023-09-30T00:38:30.217 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:38:30.229 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:38:30.229 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:38:30.288 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:38:30.292 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:38:30.405 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:38:30.647 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:31.716 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:31.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:31.716 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:38:31.716 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:31.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:31.716 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:31.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2023-09-30T00:38:32.898 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:28 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:34.653 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:34.654 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:38:34.655 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:34.655 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.5 M 2023-09-30T00:38:34.655 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 16 M 2023-09-30T00:38:34.655 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:38:36.708 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 1.7 MB/s | 3.5 MB 00:02 2023-09-30T00:38:36.709 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:36.709 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.7 MB/s | 3.5 MB 00:02 2023-09-30T00:38:36.709 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:38:36.724 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:38:36.724 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:38:36.797 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:38:36.801 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:38:37.281 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:38:37.589 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:38.060 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:38.957 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:38.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:38.958 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:38:38.958 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:38.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:38.958 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:39.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2023-09-30T00:38:40.218 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:36 ago on Sat 30 Sep 2023 12:35:04 AM UTC. 2023-09-30T00:38:41.978 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-09-30T00:38:41.979 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2023-09-30T00:38:41.980 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:41.980 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 173 k 2023-09-30T00:38:41.980 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 552 k 2023-09-30T00:38:41.980 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-09-30T00:38:42.526 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 318 kB/s | 173 kB 00:00 2023-09-30T00:38:42.527 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:42.527 INFO:teuthology.orchestra.run.smithi029.stdout:Total 317 kB/s | 173 kB 00:00 2023-09-30T00:38:42.527 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-09-30T00:38:42.540 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-09-30T00:38:42.540 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-09-30T00:38:42.600 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-09-30T00:38:42.603 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-09-30T00:38:42.745 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-09-30T00:38:42.979 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:43.384 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:44.083 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:38:44.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:44.083 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-09-30T00:38:44.083 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:44.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:44.083 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-09-30T00:38:44.298 DEBUG:teuthology.parallel:result is None 2023-09-30T00:38:44.299 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:38:44.441 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:38:44.471 INFO:teuthology.orchestra.run.smithi029.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:38:44.471 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:38:44.472 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:38:44.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:38:44.611 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:38:44.639 INFO:teuthology.orchestra.run.smithi173.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:38:44.640 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:38:44.640 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:38:44.640 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:38:44.640 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:44.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:38:44.678 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:44.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:38:44.713 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:38:44.714 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:44.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:38:44.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:38:44.828 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:44.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:38:44.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:38:44.931 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:38:44.931 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:44.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:38:44.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:38:45.041 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:45.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:38:45.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:38:45.138 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:38:45.138 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:45.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:38:45.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:38:45.248 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:45.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:38:45.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:38:45.345 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:38:45.405 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:38:45.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2023-09-30T00:38:45.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2023-09-30T00:38:45.440 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:38:45.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:38:45.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:38:45.516 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:38:45.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:38:45.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:38:45.603 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:38:45.603 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'slow request', 'overall HEALTH_', '\\(CACHE_POOL_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:38:45.603 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406704', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'], ['osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsY3dkdayRkNgoX0Qlg5cj5pGUZND1jWoYlelzq7aQsmzcxZPmCnUaXm9pGTailhMqhJwBVeBbAoLWLY3KOQhs=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrRO3VWph3/GHDDdUoVzBsEbNzIV+12yJdhkmN9+h88n62+fdyd3eNP3rnljTbH85R5E4rDMHojjQVNTbhUoTk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['but it is still running', 'slow request', 'overall HEALTH_', '\\(CACHE_POOL_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'cluster': 'ceph'}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 500']}}, {'background_exec': {'mon.a': ['while true', 'do sleep 30', 'sudo ceph osd pool set cache cache_target_full_ratio .001', 'echo cache-try-flush-evict-all', 'rados -p cache cache-try-flush-evict-all', 'sleep 5', 'echo cache-flush-evict-all', 'rados -p cache cache-flush-evict-all', 'sleep 5', 'echo remove overlay', 'sudo ceph osd tier remove-overlay base', 'sleep 20', 'done']}}, {'rados': {'clients': ['client.0'], 'max_seconds': 600, 'objects': 10000, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 400000, 'pools': ['base'], 'size': 1024}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:38:45.604 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:38:45.635 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:38:45.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:38:45.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:38:45.731 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:45.731 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:38:45.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:38:45.795 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2023-09-30T00:38:45.859 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:38:45.859 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:45.859 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 387061 Links: 1 2023-09-30T00:38:45.859 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:45.859 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:45.859 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-09-30 00:38:37.777691034 +0000 2023-09-30T00:38:45.860 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-09-30 00:33:48.835239073 +0000 2023-09-30T00:38:45.860 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-09-30 00:33:48.835239073 +0000 2023-09-30T00:38:45.860 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2023-09-30T00:38:45.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:38:45.932 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-09-30T00:38:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-09-30T00:38:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.00013093 s, 3.9 MB/s 2023-09-30T00:38:45.934 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:38:45.999 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 391997 Links: 1 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-09-30 00:38:37.777691034 +0000 2023-09-30T00:38:46.063 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-09-30 00:33:49.416229818 +0000 2023-09-30T00:38:46.064 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-09-30 00:33:49.416229818 +0000 2023-09-30T00:38:46.064 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2023-09-30T00:38:46.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:38:46.137 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-09-30T00:38:46.137 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-09-30T00:38:46.137 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000119888 s, 4.3 MB/s 2023-09-30T00:38:46.138 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:38:46.203 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2023-09-30T00:38:46.267 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:38:46.267 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:46.267 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 395534 Links: 1 2023-09-30T00:38:46.267 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:46.267 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:46.267 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-09-30 00:38:37.777691034 +0000 2023-09-30T00:38:46.268 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-09-30 00:33:50.006220419 +0000 2023-09-30T00:38:46.268 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-09-30 00:33:50.006220419 +0000 2023-09-30T00:38:46.268 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2023-09-30T00:38:46.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:38:46.341 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-09-30T00:38:46.341 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-09-30T00:38:46.341 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000109584 s, 4.7 MB/s 2023-09-30T00:38:46.342 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:38:46.406 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2023-09-30T00:38:46.470 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 395620 Links: 1 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-09-30 00:38:37.777691034 +0000 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-09-30 00:33:50.599210972 +0000 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-09-30 00:33:50.599210972 +0000 2023-09-30T00:38:46.471 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2023-09-30T00:38:46.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:38:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-09-30T00:38:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-09-30T00:38:46.545 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000136172 s, 3.8 MB/s 2023-09-30T00:38:46.547 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:38:46.612 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:46.612 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:46.612 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:38:46.636 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:38:46.636 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2023-09-30T00:38:46.699 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:38:46.699 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:46.699 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 426002 Links: 1 2023-09-30T00:38:46.699 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:46.699 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:46.699 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-09-30 00:38:20.761789661 +0000 2023-09-30T00:38:46.700 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-09-30 00:32:54.856741101 +0000 2023-09-30T00:38:46.700 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-09-30 00:32:54.856741101 +0000 2023-09-30T00:38:46.700 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-09-30T00:38:46.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:38:46.770 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-09-30T00:38:46.771 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-09-30T00:38:46.771 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000221302 s, 2.3 MB/s 2023-09-30T00:38:46.772 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:38:46.834 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2023-09-30T00:38:46.895 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:38:46.895 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 417272 Links: 1 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-09-30 00:38:20.761789661 +0000 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-09-30 00:32:55.386729856 +0000 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-09-30 00:32:55.386729856 +0000 2023-09-30T00:38:46.896 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-09-30T00:38:46.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:38:46.965 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-09-30T00:38:46.965 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-09-30T00:38:46.965 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000230749 s, 2.2 MB/s 2023-09-30T00:38:46.966 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:38:47.027 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2023-09-30T00:38:47.087 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:38:47.087 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:47.087 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 421593 Links: 1 2023-09-30T00:38:47.087 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:47.087 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:47.087 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-09-30 00:38:20.761789661 +0000 2023-09-30T00:38:47.088 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-09-30 00:32:55.905718845 +0000 2023-09-30T00:38:47.088 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-09-30 00:32:55.905718845 +0000 2023-09-30T00:38:47.088 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-09-30T00:38:47.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:38:47.155 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-09-30T00:38:47.156 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-09-30T00:38:47.156 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000224349 s, 2.3 MB/s 2023-09-30T00:38:47.157 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:38:47.216 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 426264 Links: 1 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-09-30 00:38:20.761789661 +0000 2023-09-30T00:38:47.275 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-09-30 00:32:56.437707558 +0000 2023-09-30T00:38:47.276 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-09-30 00:32:56.437707558 +0000 2023-09-30T00:38:47.276 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-09-30T00:38:47.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:38:47.341 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-09-30T00:38:47.341 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-09-30T00:38:47.341 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000238848 s, 2.1 MB/s 2023-09-30T00:38:47.342 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:38:47.403 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:47.403 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T00:38:47.403 INFO:tasks.ceph:Generating config... 2023-09-30T00:38:47.405 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T00:38:47.405 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T00:38:47.405 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T00:38:47.406 INFO:tasks.ceph:[global] ms type = async 2023-09-30T00:38:47.406 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:38:47.406 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:38:47.406 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:38:47.406 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:38:47.406 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:38:47.406 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T00:38:47.407 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T00:38:47.408 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T00:38:47.409 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T00:38:47.409 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T00:38:47.409 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T00:38:47.409 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T00:38:47.409 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T00:38:47.409 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T00:38:47.409 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:38:47.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:38:47.476 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:38:47.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:38:47.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:38:47.562 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29')] 2023-09-30T00:38:47.562 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-09-30T00:38:47.563 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:47.563 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:38:47.626 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:47.714 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 6147adfb-945f-4aac-97fa-4c6fe5df7b7a 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = octopus 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 6147adfb-945f-4aac-97fa-4c6fe5df7b7a 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2023-09-30T00:38:47.715100+0000 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:created 2023-09-30T00:38:47.715100+0000 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 15 (octopus) 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 3 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-09-30T00:38:47.715 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-09-30T00:38:47.717 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:38:47.738 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6147adfb-945f-4aac-97fa-4c6fe5df7b7a... 2023-09-30T00:38:47.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:38:47.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:38:47.832 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2023-09-30T00:38:47.832 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2023-09-30T00:38:47.832 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:38:47.832 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2023-09-30T00:38:47.832 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.832 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:38:47.833 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:38:47.834 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2023-09-30T00:38:47.835 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2023-09-30T00:38:47.836 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = 172.21.15.29 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: mon client directed command retry = 5 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: mon election default strategy = 3 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject socket failures = 5000 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: ms type = async 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 6147adfb-945f-4aac-97fa-4c6fe5df7b7a 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2023-09-30T00:38:47.837 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2023-09-30T00:38:47.838 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2023-09-30T00:38:47.839 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression algorithm = zlib 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression mode = aggressive 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore zero block detection = True 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2023-09-30T00:38:47.840 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify cached snaps = True 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify missing on start = True 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock override recovery settings = True 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue = debug_random 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue cut off = debug_random 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2023-09-30T00:38:47.841 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2023-09-30T00:38:47.842 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:38:47.843 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub interval = 300 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2023-09-30T00:38:47.844 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:38:47.845 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:38:47.845 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2023-09-30T00:38:47.845 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-09-30T00:38:47.845 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.846 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:38:47.847 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-09-30T00:38:47.848 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-09-30T00:38:47.849 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = 172.21.15.29 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: mon election default strategy = 3 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 5000 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: ms type = async 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 6147adfb-945f-4aac-97fa-4c6fe5df7b7a 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-09-30T00:38:47.850 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-09-30T00:38:47.851 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-09-30T00:38:47.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression algorithm = zlib 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression mode = aggressive 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore zero block detection = True 2023-09-30T00:38:47.853 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 20 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 20 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify cached snaps = True 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify missing on start = True 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock override recovery settings = True 2023-09-30T00:38:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue = debug_random 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue cut off = debug_random 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-09-30T00:38:47.855 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-09-30T00:38:47.856 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:38:47.857 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: mon scrub interval = 300 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:38:47.858 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-09-30T00:38:47.860 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:38:47.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:38:47.966 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:38:47.966 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:47.967 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:38:47.988 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:47.988 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:38:48.051 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:38:48.051 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:48.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:38:48.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:38:48.138 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:38:48.138 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:48.202 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:38:48.202 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:48.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:38:48.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:38:48.246 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:38:48.246 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:48.307 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:38:48.308 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:38:48.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:38:48.372 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:38:48.375 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:38:48.375 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:38:48.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:38:48.442 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:38:48.455 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:38:48.455 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T00:38:48.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:38:48.491 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:48.491 INFO:tasks.ceph:role: osd.0 2023-09-30T00:38:48.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:38:48.492 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:38:48.566 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:38:48.566 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:38:48.566 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:38:48.567 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:38:48.589 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-09-30T00:38:48.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-09-30T00:38:48.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:38:48.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:38:48.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:38:48.795 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:48.795 INFO:tasks.ceph:role: osd.1 2023-09-30T00:38:48.795 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:38:48.796 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:38:48.870 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:38:48.871 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:38:48.893 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-09-30T00:38:48.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-09-30T00:38:48.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T00:38:48.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:38:49.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:38:49.096 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:49.096 INFO:tasks.ceph:role: osd.2 2023-09-30T00:38:49.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:38:49.097 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:38:49.172 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:38:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:38:49.173 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:38:49.195 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-09-30T00:38:49.197 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-09-30T00:38:49.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T00:38:49.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:38:49.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:49.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.433+0000 7f821d240540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:38:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.433+0000 7f821d240540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:38:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.433+0000 7f821d240540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:38:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.433+0000 7f821d240540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:38:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.433+0000 7f821d240540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:38:49.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.433+0000 7f821d240540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T00:38:49.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:49.959+0000 7f821d240540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:50.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:50.719+0000 7f821d240540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:51.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:51.481+0000 7f821d240540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:52.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:38:52.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.407+0000 7f230db92540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:38:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.407+0000 7f230db92540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:38:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.407+0000 7f230db92540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:38:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.407+0000 7f230db92540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:38:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.407+0000 7f230db92540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:38:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.407+0000 7f230db92540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T00:38:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:52.933+0000 7f230db92540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:53.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:53.698+0000 7f230db92540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:54.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:54.461+0000 7f230db92540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:55.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:38:55.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:38:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.385+0000 7f00974cb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:38:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.385+0000 7f00974cb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:38:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.385+0000 7f00974cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:38:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.385+0000 7f00974cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:38:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.385+0000 7f00974cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:38:55.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.386+0000 7f00974cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T00:38:55.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:55.908+0000 7f00974cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:56.674+0000 7f00974cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:57.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-09-30T00:38:57.436+0000 7f00974cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:38:58.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:38:58.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:38:58.305 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:58.305 INFO:tasks.ceph:role: osd.3 2023-09-30T00:38:58.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:38:58.305 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:38:58.382 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:38:58.383 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:38:58.804 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-09-30T00:38:58.807 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-09-30T00:38:58.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-09-30T00:38:58.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:38:59.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T00:38:59.099 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:59.100 INFO:tasks.ceph:role: osd.4 2023-09-30T00:38:59.100 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:38:59.100 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:38:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:38:59.176 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:38:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:38:59.593 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-09-30T00:38:59.595 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-09-30T00:38:59.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-09-30T00:38:59.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T00:38:59.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T00:38:59.800 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:38:59.800 INFO:tasks.ceph:role: osd.5 2023-09-30T00:38:59.800 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:38:59.800 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:38:59.874 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:38:59.874 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:38:59.874 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:38:59.875 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:39:00.323 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-09-30T00:39:00.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-09-30T00:39:00.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-09-30T00:39:00.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T00:39:00.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:39:00.628 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:00.627+0000 7f432b265540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:39:00.631 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:00.630+0000 7f432b265540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:39:00.631 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:00.630+0000 7f432b265540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:39:00.631 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:00.630+0000 7f432b265540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:39:00.631 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:00.630+0000 7f432b265540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:39:00.637 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:00.637+0000 7f432b265540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T00:39:01.161 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:01.160+0000 7f432b265540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:01.923 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:01.922+0000 7f432b265540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:02.684 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:02.683+0000 7f432b265540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:03.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:39:03.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:39:03.600 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:03.599+0000 7fcc9a2d4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T00:39:03.601 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:03.599+0000 7fcc9a2d4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T00:39:03.601 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:03.599+0000 7fcc9a2d4540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:39:03.601 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:03.599+0000 7fcc9a2d4540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:39:03.601 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:03.599+0000 7fcc9a2d4540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:39:03.601 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:03.600+0000 7fcc9a2d4540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T00:39:04.127 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:04.126+0000 7fcc9a2d4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:04.890 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:04.889+0000 7fcc9a2d4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:05.653 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:05.653+0000 7fcc9a2d4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:06.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T00:39:06.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:39:06.570 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:06.568+0000 7fe654560540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T00:39:06.570 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:06.569+0000 7fe654560540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T00:39:06.570 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:06.569+0000 7fe654560540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:39:06.570 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:06.569+0000 7fe654560540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:39:06.571 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:06.569+0000 7fe654560540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:39:06.571 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:06.569+0000 7fe654560540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T00:39:07.094 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:07.093+0000 7fe654560540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:07.857 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:07.856+0000 7fe654560540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:08.619 INFO:teuthology.orchestra.run.smithi173.stderr:2023-09-30T00:39:08.618+0000 7fe654560540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:39:09.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T00:39:09.459 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:39:09.459 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:09.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:39:09.494 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:09.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:39:09.571 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:09.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:39:09.647 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:09.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:39:09.722 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:09.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:39:09.755 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:09.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T00:39:09.825 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:09.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T00:39:09.895 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:09.895 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:39:09.955 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:39:09.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:39:09.989 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2023-09-30T00:39:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAYbhdl3o86FhAA7V3pRb25VwgUWdgfSz45pg== 2023-09-30T00:39:09.990 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2023-09-30T00:39:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAZbhdl3hjaGRAAPMwseuQXgUlV4egYsKVbqA== 2023-09-30T00:39:09.990 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2023-09-30T00:39:09.990 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAcbhdlOoZOGBAAEYLmIMyR0dxFJpfqAahmHw== 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAfbhdlBzECFxAAbRYm4N/+BYXvi2rbJ2OPjw== 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAkbhdlQnd7JRAAzg+BJA4AKbXiFiddFF6vfQ== 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAnbhdlze/MIxAAcK6KfKo/6W8FxaOFDDekKA== 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqbhdlJj36IRAAjRChLGi9sMZV6D3Rsk31pw== 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-09-30T00:39:09.991 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAYbhdlLfZlGhAA12GC6use0cYKXXK9w+EJcg== 2023-09-30T00:39:09.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:39:10.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:39:10.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:39:10.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:39:10.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:39:10.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:39:10.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:39:10.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:39:10.494 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:39:10.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:39:10.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:39:10.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:39:10.661 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:39:10.724 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:39:10.725 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:39:10.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T00:39:10.767 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:39:10.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph config set mgr mgr_pool false --force 2023-09-30T00:39:11.004 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:39:11.004 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:39:11.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T00:39:11.007 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:39:11.007 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:39:11.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:39:11.221 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:11.221+0000 7ff00b3f61c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:39:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-09-30T00:39:11.242 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:39:11.242 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:11.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:39:11.273 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:11.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:39:11.371 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-09-30T00:39:11.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:39:11.415 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:11.415+0000 7ff00b3f61c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:39:11.442 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:11.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:39:11.474 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:11.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T00:39:11.546 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-09-30T00:39:11.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T00:39:11.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 435ed398-a081-4f06-9104-8a7555961c2a 0 2023-09-30T00:39:11.865 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:11.865+0000 7ff00b3f61c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:39:11.872 INFO:teuthology.orchestra.run.smithi173.stdout:0 2023-09-30T00:39:11.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 4e245884-87c8-497e-af9e-a856480adf1d 1 2023-09-30T00:39:12.086 INFO:teuthology.orchestra.run.smithi173.stdout:1 2023-09-30T00:39:12.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 7c620fed-a00f-40de-a548-64b3fbaf1f2e 2 2023-09-30T00:39:12.300 INFO:teuthology.orchestra.run.smithi173.stdout:2 2023-09-30T00:39:12.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new e0c974a6-9069-40ce-859f-4abcc75a5d40 3 2023-09-30T00:39:12.450 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:12.449+0000 7ff00b3f61c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:39:12.514 INFO:teuthology.orchestra.run.smithi173.stdout:3 2023-09-30T00:39:12.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 4956961b-3c7c-4268-a86b-ef77e63a3dfa 4 2023-09-30T00:39:12.604 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:12.604+0000 7ff00b3f61c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:39:12.729 INFO:teuthology.orchestra.run.smithi173.stdout:4 2023-09-30T00:39:12.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 02c3c662-187e-4ed9-a0d1-45773d3f2b16 5 2023-09-30T00:39:12.946 INFO:teuthology.orchestra.run.smithi173.stdout:5 2023-09-30T00:39:12.958 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:39:12.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T00:39:12.960 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:39:12.960 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:39:12.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T00:39:12.963 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:39:12.963 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:39:12.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T00:39:12.965 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:39:12.965 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:39:12.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T00:39:12.967 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:39:12.967 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T00:39:12.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T00:39:12.969 INFO:tasks.ceph.osd.4:Started 2023-09-30T00:39:12.969 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T00:39:12.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T00:39:12.971 INFO:tasks.ceph.osd.5:Started 2023-09-30T00:39:12.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:39:13.034 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:13.034+0000 7ff00b3f61c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:39:13.218 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:13.218 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:12.944606+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:39:13.232 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:39:13.232 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:39:13.561 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-09-30T00:39:13.560+0000 7fa74bf91540 -1 Falling back to public interface 2023-09-30T00:39:13.566 INFO:tasks.ceph.osd.4.smithi173.stderr:2023-09-30T00:39:13.565+0000 7f423a2e2540 -1 Falling back to public interface 2023-09-30T00:39:13.569 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-09-30T00:39:13.568+0000 7f637edab540 -1 Falling back to public interface 2023-09-30T00:39:13.569 INFO:tasks.ceph.osd.3.smithi173.stderr:2023-09-30T00:39:13.568+0000 7f0564a82540 -1 Falling back to public interface 2023-09-30T00:39:13.573 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-09-30T00:39:13.572+0000 7fcc2194e540 -1 Falling back to public interface 2023-09-30T00:39:13.573 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-09-30T00:39:13.573+0000 7f7c2048b540 -1 Falling back to public interface 2023-09-30T00:39:13.837 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:39:13.874 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:13.874+0000 7ff00b3f61c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:39:13.956 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:13.956+0000 7ff00b3f61c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:39:14.043 INFO:teuthology.misc.health.smithi029.stdout: 2023-09-30T00:39:14.043 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":8,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:12.944606+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:39:14.056 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-09-30T00:39:14.111 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.110+0000 7ff00b3f61c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:39:14.188 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.188+0000 7ff00b3f61c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:39:14.394 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.393+0000 7ff00b3f61c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:39:14.480 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.480+0000 7ff00b3f61c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:39:14.566 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.566+0000 7ff00b3f61c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:39:14.671 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.670+0000 7ff00b3f61c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:39:14.871 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.870+0000 7ff00b3f61c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:39:14.961 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:14.960+0000 7ff00b3f61c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:39:15.288 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:15.287+0000 7ff00b3f61c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:39:15.366 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:15.366+0000 7ff00b3f61c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:39:15.938 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:15.937+0000 7ff00b3f61c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:39:16.327 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:16.326+0000 7ff00b3f61c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:39:16.516 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:16.516+0000 7ff00b3f61c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:39:16.924 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:16.924+0000 7ff00b3f61c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:39:16.985 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-09-30T00:39:16.984+0000 7fa74bf91540 -1 osd.5 0 log_to_monitors true 2023-09-30T00:39:17.039 INFO:tasks.ceph.osd.4.smithi173.stderr:2023-09-30T00:39:17.038+0000 7f423a2e2540 -1 osd.4 0 log_to_monitors true 2023-09-30T00:39:17.043 INFO:tasks.ceph.osd.3.smithi173.stderr:2023-09-30T00:39:17.042+0000 7f0564a82540 -1 osd.3 0 log_to_monitors true 2023-09-30T00:39:17.092 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-09-30T00:39:17.092+0000 7f637edab540 -1 osd.0 0 log_to_monitors true 2023-09-30T00:39:17.093 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:17.092+0000 7ff00b3f61c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:39:17.100 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-09-30T00:39:17.099+0000 7fcc2194e540 -1 osd.1 0 log_to_monitors true 2023-09-30T00:39:17.104 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-09-30T00:39:17.104+0000 7f7c2048b540 -1 osd.2 0 log_to_monitors true 2023-09-30T00:39:17.266 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:17.266+0000 7ff00b3f61c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:39:17.381 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:17.378+0000 7ff00b3f61c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:39:17.422 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:17.422+0000 7fef9a8a0700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:39:17.422 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:17.422+0000 7fef9a8a0700 -1 pg_autoscaler.serve: 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:17.422+0000 7fef9a8a0700 -1 Traceback (most recent call last): 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr: self._maybe_adjust() 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr: if self.noautoscale: 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:39:17.424 INFO:tasks.ceph.mgr.x.smithi029.stderr: 2023-09-30T00:39:18.047 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-09-30T00:39:18.046+0000 7fa742b96700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T00:39:18.051 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-09-30T00:39:18.050+0000 7fa73d991700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:39:18.400 INFO:tasks.ceph.mgr.x.smithi029.stderr:2023-09-30T00:39:18.399+0000 7fefcfada700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-30T00:39:19.052 INFO:tasks.ceph.osd.4.smithi173.stderr:2023-09-30T00:39:19.051+0000 7f4230ee7700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T00:39:19.052 INFO:tasks.ceph.osd.3.smithi173.stderr:2023-09-30T00:39:19.051+0000 7f055b687700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:39:19.053 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-09-30T00:39:19.052+0000 7f63759b0700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:39:19.053 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-09-30T00:39:19.053+0000 7fcc18553700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:39:19.054 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-09-30T00:39:19.054+0000 7f7c17090700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:39:19.056 INFO:tasks.ceph.osd.4.smithi173.stderr:2023-09-30T00:39:19.055+0000 7f422bce2700 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:39:19.056 INFO:tasks.ceph.osd.3.smithi173.stderr:2023-09-30T00:39:19.056+0000 7f0556482700 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:39:19.058 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-09-30T00:39:19.058+0000 7f63707ab700 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:39:19.058 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-09-30T00:39:19.058+0000 7fcc1334e700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:39:19.059 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-09-30T00:39:19.059+0000 7f7c11e8b700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:39:19.400 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-09-30T00:39:19.400+0000 7f0431f0c700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:39:20.661 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:39:21.006 INFO:teuthology.misc.health.smithi029.stdout: 2023-09-30T00:39:21.006 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":12,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:20.047161+0000","last_up_change":"2023-09-30T00:39:20.047161+0000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":114351},{"type":"v1","addr":"172.21.15.29:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":114351},{"type":"v1","addr":"172.21.15.29:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":114351},{"type":"v1","addr":"172.21.15.29:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":114351},{"type":"v1","addr":"172.21.15.29:6805","nonce":114351}]},"public_addr":"172.21.15.29:6801/114351","cluster_addr":"172.21.15.29:6803/114351","heartbeat_back_addr":"172.21.15.29:6807/114351","heartbeat_front_addr":"172.21.15.29:6805/114351","state":["exists","up"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":114350},{"type":"v1","addr":"172.21.15.29:6810","nonce":114350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":114350},{"type":"v1","addr":"172.21.15.29:6815","nonce":114350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":114350},{"type":"v1","addr":"172.21.15.29:6823","nonce":114350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":114350},{"type":"v1","addr":"172.21.15.29:6820","nonce":114350}]},"public_addr":"172.21.15.29:6810/114350","cluster_addr":"172.21.15.29:6815/114350","heartbeat_back_addr":"172.21.15.29:6823/114350","heartbeat_front_addr":"172.21.15.29:6820/114350","state":["exists","up"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":114352},{"type":"v1","addr":"172.21.15.29:6811","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":114352},{"type":"v1","addr":"172.21.15.29:6814","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":114352},{"type":"v1","addr":"172.21.15.29:6821","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":114352},{"type":"v1","addr":"172.21.15.29:6817","nonce":114352}]},"public_addr":"172.21.15.29:6811/114352","cluster_addr":"172.21.15.29:6814/114352","heartbeat_back_addr":"172.21.15.29:6821/114352","heartbeat_front_addr":"172.21.15.29:6817/114352","state":["exists","up"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":142501},{"type":"v1","addr":"172.21.15.173:6817","nonce":142501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":142501},{"type":"v1","addr":"172.21.15.173:6819","nonce":142501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":142501},{"type":"v1","addr":"172.21.15.173:6823","nonce":142501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":142501},{"type":"v1","addr":"172.21.15.173:6821","nonce":142501}]},"public_addr":"172.21.15.173:6817/142501","cluster_addr":"172.21.15.173:6819/142501","heartbeat_back_addr":"172.21.15.173:6823/142501","heartbeat_front_addr":"172.21.15.173:6821/142501","state":["exists","up"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":142502},{"type":"v1","addr":"172.21.15.173:6809","nonce":142502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":142502},{"type":"v1","addr":"172.21.15.173:6811","nonce":142502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":142502},{"type":"v1","addr":"172.21.15.173:6815","nonce":142502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":142502},{"type":"v1","addr":"172.21.15.173:6813","nonce":142502}]},"public_addr":"172.21.15.173:6809/142502","cluster_addr":"172.21.15.173:6811/142502","heartbeat_back_addr":"172.21.15.173:6815/142502","heartbeat_front_addr":"172.21.15.173:6813/142502","state":["exists","up"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":142503},{"type":"v1","addr":"172.21.15.173:6801","nonce":142503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":142503},{"type":"v1","addr":"172.21.15.173:6803","nonce":142503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":142503},{"type":"v1","addr":"172.21.15.173:6807","nonce":142503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":142503},{"type":"v1","addr":"172.21.15.173:6805","nonce":142503}]},"public_addr":"172.21.15.173:6801/142503","cluster_addr":"172.21.15.173:6803/142503","heartbeat_back_addr":"172.21.15.173:6807/142503","heartbeat_front_addr":"172.21.15.173:6805/142503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:17.980520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:39:21.020 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-09-30T00:39:21.020 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:39:21.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:39:22.061 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2023-09-30T00:39:22.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:39:23.066 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:39:23.079 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:39:23.080 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:39:23.080 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:39:23.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:39:23.081 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:39:23.485 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:23.498 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":114107},{"type":"v1","addr":"172.21.15.29:6825","nonce":114107}]},"active_addr":"172.21.15.29:6825/114107","active_change":"2023-09-30T00:39:17.392716+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":4015651326}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2771900694}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2164222524}]}]} 2023-09-30T00:39:23.500 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:39:23.501 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:39:23.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:39:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:23.844 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:23.055996+0000","last_up_change":"2023-09-30T00:39:20.047161+0000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:39:21.399773+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":114351},{"type":"v1","addr":"172.21.15.29:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":114351},{"type":"v1","addr":"172.21.15.29:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":114351},{"type":"v1","addr":"172.21.15.29:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":114351},{"type":"v1","addr":"172.21.15.29:6805","nonce":114351}]},"public_addr":"172.21.15.29:6801/114351","cluster_addr":"172.21.15.29:6803/114351","heartbeat_back_addr":"172.21.15.29:6807/114351","heartbeat_front_addr":"172.21.15.29:6805/114351","state":["exists","up"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":114350},{"type":"v1","addr":"172.21.15.29:6810","nonce":114350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":114350},{"type":"v1","addr":"172.21.15.29:6815","nonce":114350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":114350},{"type":"v1","addr":"172.21.15.29:6823","nonce":114350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":114350},{"type":"v1","addr":"172.21.15.29:6820","nonce":114350}]},"public_addr":"172.21.15.29:6810/114350","cluster_addr":"172.21.15.29:6815/114350","heartbeat_back_addr":"172.21.15.29:6823/114350","heartbeat_front_addr":"172.21.15.29:6820/114350","state":["exists","up"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":114352},{"type":"v1","addr":"172.21.15.29:6811","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":114352},{"type":"v1","addr":"172.21.15.29:6814","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":114352},{"type":"v1","addr":"172.21.15.29:6821","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":114352},{"type":"v1","addr":"172.21.15.29:6817","nonce":114352}]},"public_addr":"172.21.15.29:6811/114352","cluster_addr":"172.21.15.29:6814/114352","heartbeat_back_addr":"172.21.15.29:6821/114352","heartbeat_front_addr":"172.21.15.29:6817/114352","state":["exists","up"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":142501},{"type":"v1","addr":"172.21.15.173:6817","nonce":142501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":142501},{"type":"v1","addr":"172.21.15.173:6819","nonce":142501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":142501},{"type":"v1","addr":"172.21.15.173:6823","nonce":142501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":142501},{"type":"v1","addr":"172.21.15.173:6821","nonce":142501}]},"public_addr":"172.21.15.173:6817/142501","cluster_addr":"172.21.15.173:6819/142501","heartbeat_back_addr":"172.21.15.173:6823/142501","heartbeat_front_addr":"172.21.15.173:6821/142501","state":["exists","up"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":142502},{"type":"v1","addr":"172.21.15.173:6809","nonce":142502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":142502},{"type":"v1","addr":"172.21.15.173:6811","nonce":142502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":142502},{"type":"v1","addr":"172.21.15.173:6815","nonce":142502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":142502},{"type":"v1","addr":"172.21.15.173:6813","nonce":142502}]},"public_addr":"172.21.15.173:6809/142502","cluster_addr":"172.21.15.173:6811/142502","heartbeat_back_addr":"172.21.15.173:6815/142502","heartbeat_front_addr":"172.21.15.173:6813/142502","state":["exists","up"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":142503},{"type":"v1","addr":"172.21.15.173:6801","nonce":142503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":142503},{"type":"v1","addr":"172.21.15.173:6803","nonce":142503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":142503},{"type":"v1","addr":"172.21.15.173:6807","nonce":142503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":142503},{"type":"v1","addr":"172.21.15.173:6805","nonce":142503}]},"public_addr":"172.21.15.173:6801/142503","cluster_addr":"172.21.15.173:6803/142503","heartbeat_back_addr":"172.21.15.173:6807/142503","heartbeat_front_addr":"172.21.15.173:6805/142503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.108600+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.127036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.156519+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.003755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.016418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:17.980520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:39:23.859 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:39:23.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:39:24.207 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:24.207 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:24.060378+0000","last_up_change":"2023-09-30T00:39:20.047161+0000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:39:21.399773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":114351},{"type":"v1","addr":"172.21.15.29:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":114351},{"type":"v1","addr":"172.21.15.29:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":114351},{"type":"v1","addr":"172.21.15.29:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":114351},{"type":"v1","addr":"172.21.15.29:6805","nonce":114351}]},"public_addr":"172.21.15.29:6801/114351","cluster_addr":"172.21.15.29:6803/114351","heartbeat_back_addr":"172.21.15.29:6807/114351","heartbeat_front_addr":"172.21.15.29:6805/114351","state":["exists","up"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":114350},{"type":"v1","addr":"172.21.15.29:6810","nonce":114350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":114350},{"type":"v1","addr":"172.21.15.29:6815","nonce":114350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":114350},{"type":"v1","addr":"172.21.15.29:6823","nonce":114350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":114350},{"type":"v1","addr":"172.21.15.29:6820","nonce":114350}]},"public_addr":"172.21.15.29:6810/114350","cluster_addr":"172.21.15.29:6815/114350","heartbeat_back_addr":"172.21.15.29:6823/114350","heartbeat_front_addr":"172.21.15.29:6820/114350","state":["exists","up"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":114352},{"type":"v1","addr":"172.21.15.29:6811","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":114352},{"type":"v1","addr":"172.21.15.29:6814","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":114352},{"type":"v1","addr":"172.21.15.29:6821","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":114352},{"type":"v1","addr":"172.21.15.29:6817","nonce":114352}]},"public_addr":"172.21.15.29:6811/114352","cluster_addr":"172.21.15.29:6814/114352","heartbeat_back_addr":"172.21.15.29:6821/114352","heartbeat_front_addr":"172.21.15.29:6817/114352","state":["exists","up"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":142501},{"type":"v1","addr":"172.21.15.173:6817","nonce":142501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":142501},{"type":"v1","addr":"172.21.15.173:6819","nonce":142501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":142501},{"type":"v1","addr":"172.21.15.173:6823","nonce":142501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":142501},{"type":"v1","addr":"172.21.15.173:6821","nonce":142501}]},"public_addr":"172.21.15.173:6817/142501","cluster_addr":"172.21.15.173:6819/142501","heartbeat_back_addr":"172.21.15.173:6823/142501","heartbeat_front_addr":"172.21.15.173:6821/142501","state":["exists","up"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":142502},{"type":"v1","addr":"172.21.15.173:6809","nonce":142502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":142502},{"type":"v1","addr":"172.21.15.173:6811","nonce":142502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":142502},{"type":"v1","addr":"172.21.15.173:6815","nonce":142502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":142502},{"type":"v1","addr":"172.21.15.173:6813","nonce":142502}]},"public_addr":"172.21.15.173:6809/142502","cluster_addr":"172.21.15.173:6811/142502","heartbeat_back_addr":"172.21.15.173:6815/142502","heartbeat_front_addr":"172.21.15.173:6813/142502","state":["exists","up"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":142503},{"type":"v1","addr":"172.21.15.173:6801","nonce":142503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":142503},{"type":"v1","addr":"172.21.15.173:6803","nonce":142503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":142503},{"type":"v1","addr":"172.21.15.173:6807","nonce":142503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":142503},{"type":"v1","addr":"172.21.15.173:6805","nonce":142503}]},"public_addr":"172.21.15.173:6801/142503","cluster_addr":"172.21.15.173:6803/142503","heartbeat_back_addr":"172.21.15.173:6807/142503","heartbeat_front_addr":"172.21.15.173:6805/142503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.108600+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.127036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.156519+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.003755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.016418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:17.980520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:39:24.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:39:24.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:39:24.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:39:24.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:39:24.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T00:39:24.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T00:39:24.446 INFO:teuthology.orchestra.run.smithi029.stdout:51539607555 2023-09-30T00:39:24.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:39:24.447 INFO:teuthology.orchestra.run.smithi029.stdout:51539607554 2023-09-30T00:39:24.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:39:24.486 INFO:teuthology.orchestra.run.smithi029.stdout:51539607554 2023-09-30T00:39:24.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:39:24.497 INFO:teuthology.orchestra.run.smithi029.stdout:47244640260 2023-09-30T00:39:24.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:39:24.505 INFO:teuthology.orchestra.run.smithi029.stdout:51539607554 2023-09-30T00:39:24.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:39:24.506 INFO:teuthology.orchestra.run.smithi029.stdout:51539607554 2023-09-30T00:39:24.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:39:24.935 INFO:teuthology.orchestra.run.smithi029.stdout:51539607553 2023-09-30T00:39:24.955 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.1 2023-09-30T00:39:24.989 INFO:teuthology.orchestra.run.smithi029.stdout:51539607553 2023-09-30T00:39:25.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.2 2023-09-30T00:39:25.041 INFO:teuthology.orchestra.run.smithi029.stdout:51539607554 2023-09-30T00:39:25.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.4 2023-09-30T00:39:25.066 INFO:teuthology.orchestra.run.smithi029.stdout:51539607553 2023-09-30T00:39:25.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.0 2023-09-30T00:39:25.119 INFO:teuthology.orchestra.run.smithi029.stdout:47244640258 2023-09-30T00:39:25.129 INFO:teuthology.orchestra.run.smithi029.stdout:51539607553 2023-09-30T00:39:25.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2023-09-30T00:39:25.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.3 2023-09-30T00:39:25.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:39:26.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:39:26.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:39:26.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:39:26.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:39:26.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:39:26.512 INFO:teuthology.orchestra.run.smithi029.stdout:51539607555 2023-09-30T00:39:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:51539607555 2023-09-30T00:39:26.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607555 for osd.1 2023-09-30T00:39:26.536 DEBUG:teuthology.parallel:result is None 2023-09-30T00:39:26.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607555 for osd.2 2023-09-30T00:39:26.542 DEBUG:teuthology.parallel:result is None 2023-09-30T00:39:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:47244640260 2023-09-30T00:39:26.572 INFO:teuthology.orchestra.run.smithi029.stdout:51539607555 2023-09-30T00:39:26.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2023-09-30T00:39:26.574 DEBUG:teuthology.parallel:result is None 2023-09-30T00:39:26.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607555 for osd.3 2023-09-30T00:39:26.588 DEBUG:teuthology.parallel:result is None 2023-09-30T00:39:26.604 INFO:teuthology.orchestra.run.smithi029.stdout:51539607556 2023-09-30T00:39:26.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607556 for osd.4 2023-09-30T00:39:26.619 DEBUG:teuthology.parallel:result is None 2023-09-30T00:39:26.631 INFO:teuthology.orchestra.run.smithi029.stdout:51539607555 2023-09-30T00:39:26.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607555 for osd.0 2023-09-30T00:39:26.648 DEBUG:teuthology.parallel:result is None 2023-09-30T00:39:26.648 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:39:26.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:39:27.031 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:39:27.046 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T00:39:25.398234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30536,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":29696,"kb_avail":566200504,"statfs":{"total":579820584960,"available":579789316096,"internally_reserved":0,"allocated":811008,"data_stored":245838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30408704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064666+0000","last_change":"2023-09-30T00:39:23.063388+0000","last_active":"2023-09-30T00:39:23.064666+0000","last_peered":"2023-09-30T00:39:23.064666+0000","last_clean":"2023-09-30T00:39:23.064666+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:39:23.064666+0000","last_undegraded":"2023-09-30T00:39:23.064666+0000","last_fullsized":"2023-09-30T00:39:23.064666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:07.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064544+0000","last_change":"2023-09-30T00:39:23.063310+0000","last_active":"2023-09-30T00:39:23.064544+0000","last_peered":"2023-09-30T00:39:23.064544+0000","last_clean":"2023-09-30T00:39:23.064544+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:39:23.064544+0000","last_undegraded":"2023-09-30T00:39:23.064544+0000","last_fullsized":"2023-09-30T00:39:23.064544+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065449+0000","last_change":"2023-09-30T00:39:23.064396+0000","last_active":"2023-09-30T00:39:23.065449+0000","last_peered":"2023-09-30T00:39:23.065449+0000","last_clean":"2023-09-30T00:39:23.065449+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:39:23.065449+0000","last_undegraded":"2023-09-30T00:39:23.065449+0000","last_fullsized":"2023-09-30T00:39:23.065449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:12:59.211492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.899738+0000","last_change":"2023-09-30T00:39:23.899211+0000","last_active":"2023-09-30T00:39:23.899738+0000","last_peered":"2023-09-30T00:39:23.899738+0000","last_clean":"2023-09-30T00:39:23.899738+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:39:23.899738+0000","last_undegraded":"2023-09-30T00:39:23.899738+0000","last_fullsized":"2023-09-30T00:39:23.899738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:04:35.280729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064930+0000","last_change":"2023-09-30T00:39:23.062713+0000","last_active":"2023-09-30T00:39:23.064930+0000","last_peered":"2023-09-30T00:39:23.064930+0000","last_clean":"2023-09-30T00:39:23.064930+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:39:23.064930+0000","last_undegraded":"2023-09-30T00:39:23.064930+0000","last_fullsized":"2023-09-30T00:39:23.064930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:58.102938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065941+0000","last_change":"2023-09-30T00:39:23.065479+0000","last_active":"2023-09-30T00:39:23.065941+0000","last_peered":"2023-09-30T00:39:23.065941+0000","last_clean":"2023-09-30T00:39:23.065941+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:39:23.065941+0000","last_undegraded":"2023-09-30T00:39:23.065941+0000","last_fullsized":"2023-09-30T00:39:23.065941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:21:46.679489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065380+0000","last_change":"2023-09-30T00:39:23.064537+0000","last_active":"2023-09-30T00:39:23.065380+0000","last_peered":"2023-09-30T00:39:23.065380+0000","last_clean":"2023-09-30T00:39:23.065380+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:39:23.065380+0000","last_undegraded":"2023-09-30T00:39:23.065380+0000","last_fullsized":"2023-09-30T00:39:23.065380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:26:05.208826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065342+0000","last_change":"2023-09-30T00:39:23.064523+0000","last_active":"2023-09-30T00:39:23.065342+0000","last_peered":"2023-09-30T00:39:23.065342+0000","last_clean":"2023-09-30T00:39:23.065342+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:39:23.065342+0000","last_undegraded":"2023-09-30T00:39:23.065342+0000","last_fullsized":"2023-09-30T00:39:23.065342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:21:29.969487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607555,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":1,"up_from":12,"seq":51539607555,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:39:27.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:39:27.392 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:27.394 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:39:27.407 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T00:39:25.398234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30536,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":29696,"kb_avail":566200504,"statfs":{"total":579820584960,"available":579789316096,"internally_reserved":0,"allocated":811008,"data_stored":245838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30408704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064666+0000","last_change":"2023-09-30T00:39:23.063388+0000","last_active":"2023-09-30T00:39:23.064666+0000","last_peered":"2023-09-30T00:39:23.064666+0000","last_clean":"2023-09-30T00:39:23.064666+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:39:23.064666+0000","last_undegraded":"2023-09-30T00:39:23.064666+0000","last_fullsized":"2023-09-30T00:39:23.064666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:07.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064544+0000","last_change":"2023-09-30T00:39:23.063310+0000","last_active":"2023-09-30T00:39:23.064544+0000","last_peered":"2023-09-30T00:39:23.064544+0000","last_clean":"2023-09-30T00:39:23.064544+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:39:23.064544+0000","last_undegraded":"2023-09-30T00:39:23.064544+0000","last_fullsized":"2023-09-30T00:39:23.064544+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065449+0000","last_change":"2023-09-30T00:39:23.064396+0000","last_active":"2023-09-30T00:39:23.065449+0000","last_peered":"2023-09-30T00:39:23.065449+0000","last_clean":"2023-09-30T00:39:23.065449+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:39:23.065449+0000","last_undegraded":"2023-09-30T00:39:23.065449+0000","last_fullsized":"2023-09-30T00:39:23.065449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:12:59.211492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.899738+0000","last_change":"2023-09-30T00:39:23.899211+0000","last_active":"2023-09-30T00:39:23.899738+0000","last_peered":"2023-09-30T00:39:23.899738+0000","last_clean":"2023-09-30T00:39:23.899738+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:39:23.899738+0000","last_undegraded":"2023-09-30T00:39:23.899738+0000","last_fullsized":"2023-09-30T00:39:23.899738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:04:35.280729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064930+0000","last_change":"2023-09-30T00:39:23.062713+0000","last_active":"2023-09-30T00:39:23.064930+0000","last_peered":"2023-09-30T00:39:23.064930+0000","last_clean":"2023-09-30T00:39:23.064930+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:39:23.064930+0000","last_undegraded":"2023-09-30T00:39:23.064930+0000","last_fullsized":"2023-09-30T00:39:23.064930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:58.102938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065941+0000","last_change":"2023-09-30T00:39:23.065479+0000","last_active":"2023-09-30T00:39:23.065941+0000","last_peered":"2023-09-30T00:39:23.065941+0000","last_clean":"2023-09-30T00:39:23.065941+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:39:23.065941+0000","last_undegraded":"2023-09-30T00:39:23.065941+0000","last_fullsized":"2023-09-30T00:39:23.065941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:21:46.679489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065380+0000","last_change":"2023-09-30T00:39:23.064537+0000","last_active":"2023-09-30T00:39:23.065380+0000","last_peered":"2023-09-30T00:39:23.065380+0000","last_clean":"2023-09-30T00:39:23.065380+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:39:23.065380+0000","last_undegraded":"2023-09-30T00:39:23.065380+0000","last_fullsized":"2023-09-30T00:39:23.065380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:26:05.208826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065342+0000","last_change":"2023-09-30T00:39:23.064523+0000","last_active":"2023-09-30T00:39:23.065342+0000","last_peered":"2023-09-30T00:39:23.065342+0000","last_clean":"2023-09-30T00:39:23.065342+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:39:23.065342+0000","last_undegraded":"2023-09-30T00:39:23.065342+0000","last_fullsized":"2023-09-30T00:39:23.065342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:21:29.969487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607555,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":1,"up_from":12,"seq":51539607555,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":12,"seq":51539607555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:39:27.408 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:39:27.408 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:39:27.408 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:39:27.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:27.795 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:27.796 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:30.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:31.205 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:31.205 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:34.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:34.613 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:34.613 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:37.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:38.028 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:38.028 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:41.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:41.442 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:41.442 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:44.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:44.860 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:44.860 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:47.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:48.274 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:51.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:51.687 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:51.687 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:54.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:55.097 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:39:58.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:39:58.515 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:39:58.515 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:01.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:01.930 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:01.931 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:04.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:05.345 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:05.345 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:08.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:08.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:08.762 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:11.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:12.174 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:12.174 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:15.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:15.588 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:15.588 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:18.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:18.997 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:18.997 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:22.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:22.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:22.409 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:25.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:25.819 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:25.819 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:28.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:29.230 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:29.231 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:32.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:32.646 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:32.646 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:35.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:36.061 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:36.062 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:39.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:39.475 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:42.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:42.881 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:42.881 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:45.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:46.293 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:46.293 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:49.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:49.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:49.706 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:52.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:53.121 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:53.121 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:56.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:56.534 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:56.534 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:40:59.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:40:59.948 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:40:59.949 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:02.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:03.356 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:06.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:06.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:06.768 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:09.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:10.176 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:13.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:13.588 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:13.589 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:16.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:17.000 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:17.000 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:20.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:20.416 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:20.416 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:23.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:23.829 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:26.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:27.240 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:27.241 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:30.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:30.650 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:30.650 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:33.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:34.066 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:34.066 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:37.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:37.477 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:37.478 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:40.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:40.892 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:43.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:44.302 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:44.302 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:47.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:47.717 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:50.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:51.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:51.128 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:54.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:54.556 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:54.556 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:41:57.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:41:57.970 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:41:57.971 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:00.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:01.379 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:01.379 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:04.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:04.795 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:04.795 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:07.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:08.212 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:08.213 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:11.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:11.624 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:11.624 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:14.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:15.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:15.035 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:18.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:18.447 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:18.447 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:21.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:21.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:21.864 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:24.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:25.278 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:25.278 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:28.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:28.688 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:28.689 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:31.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:32.098 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:32.099 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:35.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:35.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:35.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:38.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:38.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:38.935 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:41.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:42.346 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:42.346 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:45.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:45.761 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:45.761 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:48.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:49.177 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:52.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:52.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:52.589 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:55.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:56.004 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:56.004 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:42:59.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:42:59.415 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:42:59.415 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:02.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:02.826 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:02.826 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:05.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:06.238 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:09.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:09.653 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:09.654 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:12.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:13.066 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:13.066 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:16.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:16.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:16.480 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:19.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:19.899 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:19.899 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:22.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:23.311 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:23.312 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:26.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:26.730 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:26.730 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:29.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:30.143 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:30.144 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:33.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:33.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:33.554 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:36.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:36.969 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:36.969 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:39.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:40.378 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:40.378 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:43.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:43.793 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:46.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:47.203 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:47.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:50.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:50.621 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:50.622 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:53.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:54.037 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:54.037 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:43:57.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:43:57.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:43:57.451 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:00.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:00.862 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:00.862 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:03.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:04.276 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:07.292 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:07.695 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:07.695 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:10.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:11.107 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:11.107 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:14.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:17.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:17.937 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:17.937 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:20.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:21.346 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:21.346 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:24.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:24.755 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:24.755 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:27.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:28.166 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:28.166 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:28.181 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:44:28.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:28.527 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:28.527 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:24.060378+0000","last_up_change":"2023-09-30T00:39:20.047161+0000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:39:21.399773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":114351},{"type":"v1","addr":"172.21.15.29:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":114351},{"type":"v1","addr":"172.21.15.29:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":114351},{"type":"v1","addr":"172.21.15.29:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":114351},{"type":"v1","addr":"172.21.15.29:6805","nonce":114351}]},"public_addr":"172.21.15.29:6801/114351","cluster_addr":"172.21.15.29:6803/114351","heartbeat_back_addr":"172.21.15.29:6807/114351","heartbeat_front_addr":"172.21.15.29:6805/114351","state":["exists","up"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":114350},{"type":"v1","addr":"172.21.15.29:6810","nonce":114350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":114350},{"type":"v1","addr":"172.21.15.29:6815","nonce":114350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":114350},{"type":"v1","addr":"172.21.15.29:6823","nonce":114350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":114350},{"type":"v1","addr":"172.21.15.29:6820","nonce":114350}]},"public_addr":"172.21.15.29:6810/114350","cluster_addr":"172.21.15.29:6815/114350","heartbeat_back_addr":"172.21.15.29:6823/114350","heartbeat_front_addr":"172.21.15.29:6820/114350","state":["exists","up"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":114352},{"type":"v1","addr":"172.21.15.29:6811","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":114352},{"type":"v1","addr":"172.21.15.29:6814","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":114352},{"type":"v1","addr":"172.21.15.29:6821","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":114352},{"type":"v1","addr":"172.21.15.29:6817","nonce":114352}]},"public_addr":"172.21.15.29:6811/114352","cluster_addr":"172.21.15.29:6814/114352","heartbeat_back_addr":"172.21.15.29:6821/114352","heartbeat_front_addr":"172.21.15.29:6817/114352","state":["exists","up"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":142501},{"type":"v1","addr":"172.21.15.173:6817","nonce":142501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":142501},{"type":"v1","addr":"172.21.15.173:6819","nonce":142501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":142501},{"type":"v1","addr":"172.21.15.173:6823","nonce":142501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":142501},{"type":"v1","addr":"172.21.15.173:6821","nonce":142501}]},"public_addr":"172.21.15.173:6817/142501","cluster_addr":"172.21.15.173:6819/142501","heartbeat_back_addr":"172.21.15.173:6823/142501","heartbeat_front_addr":"172.21.15.173:6821/142501","state":["exists","up"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":142502},{"type":"v1","addr":"172.21.15.173:6809","nonce":142502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":142502},{"type":"v1","addr":"172.21.15.173:6811","nonce":142502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":142502},{"type":"v1","addr":"172.21.15.173:6815","nonce":142502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":142502},{"type":"v1","addr":"172.21.15.173:6813","nonce":142502}]},"public_addr":"172.21.15.173:6809/142502","cluster_addr":"172.21.15.173:6811/142502","heartbeat_back_addr":"172.21.15.173:6815/142502","heartbeat_front_addr":"172.21.15.173:6813/142502","state":["exists","up"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":142503},{"type":"v1","addr":"172.21.15.173:6801","nonce":142503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":142503},{"type":"v1","addr":"172.21.15.173:6803","nonce":142503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":142503},{"type":"v1","addr":"172.21.15.173:6807","nonce":142503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":142503},{"type":"v1","addr":"172.21.15.173:6805","nonce":142503}]},"public_addr":"172.21.15.173:6801/142503","cluster_addr":"172.21.15.173:6803/142503","heartbeat_back_addr":"172.21.15.173:6807/142503","heartbeat_front_addr":"172.21.15.173:6805/142503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.108600+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.127036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.156519+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.003755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.016418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:17.980520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:28.542 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:44:28.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:28.883 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:28.885 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:44:28.897 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-30T00:44:27.457239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30624,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":29760,"kb_avail":566200416,"statfs":{"total":579820584960,"available":579789225984,"internally_reserved":0,"allocated":835584,"data_stored":259296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30474240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064666+0000","last_change":"2023-09-30T00:39:23.063388+0000","last_active":"2023-09-30T00:39:23.064666+0000","last_peered":"2023-09-30T00:39:23.064666+0000","last_clean":"2023-09-30T00:39:23.064666+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:39:23.064666+0000","last_undegraded":"2023-09-30T00:39:23.064666+0000","last_fullsized":"2023-09-30T00:39:23.064666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:07.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064544+0000","last_change":"2023-09-30T00:39:23.063310+0000","last_active":"2023-09-30T00:39:23.064544+0000","last_peered":"2023-09-30T00:39:23.064544+0000","last_clean":"2023-09-30T00:39:23.064544+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:39:23.064544+0000","last_undegraded":"2023-09-30T00:39:23.064544+0000","last_fullsized":"2023-09-30T00:39:23.064544+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065449+0000","last_change":"2023-09-30T00:39:23.064396+0000","last_active":"2023-09-30T00:39:23.065449+0000","last_peered":"2023-09-30T00:39:23.065449+0000","last_clean":"2023-09-30T00:39:23.065449+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:39:23.065449+0000","last_undegraded":"2023-09-30T00:39:23.065449+0000","last_fullsized":"2023-09-30T00:39:23.065449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:12:59.211492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.899738+0000","last_change":"2023-09-30T00:39:23.899211+0000","last_active":"2023-09-30T00:39:23.899738+0000","last_peered":"2023-09-30T00:39:23.899738+0000","last_clean":"2023-09-30T00:39:23.899738+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:39:23.899738+0000","last_undegraded":"2023-09-30T00:39:23.899738+0000","last_fullsized":"2023-09-30T00:39:23.899738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:04:35.280729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064930+0000","last_change":"2023-09-30T00:39:23.062713+0000","last_active":"2023-09-30T00:39:23.064930+0000","last_peered":"2023-09-30T00:39:23.064930+0000","last_clean":"2023-09-30T00:39:23.064930+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:39:23.064930+0000","last_undegraded":"2023-09-30T00:39:23.064930+0000","last_fullsized":"2023-09-30T00:39:23.064930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:58.102938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065941+0000","last_change":"2023-09-30T00:39:23.065479+0000","last_active":"2023-09-30T00:39:23.065941+0000","last_peered":"2023-09-30T00:39:23.065941+0000","last_clean":"2023-09-30T00:39:23.065941+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:39:23.065941+0000","last_undegraded":"2023-09-30T00:39:23.065941+0000","last_fullsized":"2023-09-30T00:39:23.065941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:21:46.679489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065380+0000","last_change":"2023-09-30T00:39:23.064537+0000","last_active":"2023-09-30T00:39:23.065380+0000","last_peered":"2023-09-30T00:39:23.065380+0000","last_clean":"2023-09-30T00:39:23.065380+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:39:23.065380+0000","last_undegraded":"2023-09-30T00:39:23.065380+0000","last_fullsized":"2023-09-30T00:39:23.065380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:26:05.208826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065342+0000","last_change":"2023-09-30T00:39:23.064523+0000","last_active":"2023-09-30T00:39:23.065342+0000","last_peered":"2023-09-30T00:39:23.065342+0000","last_clean":"2023-09-30T00:39:23.065342+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:39:23.065342+0000","last_undegraded":"2023-09-30T00:39:23.065342+0000","last_fullsized":"2023-09-30T00:39:23.065342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:21:29.969487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.60699999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.499,"15min":0.497},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65800000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.71599999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63500000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.63800000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40799999999999997},"min":{"1min":0.29599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.56799999999999995,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.40500000000000003},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53400000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.435,"15min":0.442},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.505}]}]},{"osd":3,"up_from":12,"seq":51539607615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47699999999999998},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60399999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.39000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":1.046,"15min":1.046},"last":0.52300000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71099999999999997,"5min":0.747,"15min":0.747},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.73999999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.499,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.379},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53200000000000003,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.32700000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.436,"15min":0.434},"min":{"1min":0.313,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59499999999999997,"5min":0.625,"15min":0.625},"last":0.48799999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.221},{"interface":"front","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.56200000000000006,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.50700000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47399999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74399999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.748,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.48199999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.56299999999999994},"min":{"1min":0.40899999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70299999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.42299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.74299999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.45100000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.41699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.45200000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.379}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40400000000000003,"15min":0.432},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.46300000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.40200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.46899999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.44800000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74299999999999999,"5min":1.155,"15min":1.155},"last":0.54700000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.625,"15min":0.63300000000000001},"min":{"1min":0.47899999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.77200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.73099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.46200000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.69399999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56399999999999995}]}]},{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.57899999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.372,"5min":0.37,"15min":0.37},"max":{"1min":0.61199999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.497}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.372},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.375},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.621,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.57599999999999996}]}]},{"osd":0,"up_from":12,"seq":51539607615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.34899999999999998,"15min":0.33900000000000002},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.38500000000000001},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.54800000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38700000000000001,"15min":0.38100000000000001},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53300000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.624,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.501,"5min":0.437,"15min":0.437},"max":{"1min":0.78500000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.443},{"interface":"front","average":{"1min":0.624,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.5,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86599999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59899999999999998,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.61699999999999999},"min":{"1min":0.47399999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.376},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.624,"15min":0.621},"min":{"1min":0.56000000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:28.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:29.241 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:44:29.256 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2023-09-30T00:44:27.457239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30624,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":29760,"kb_avail":566200416,"statfs":{"total":579820584960,"available":579789225984,"internally_reserved":0,"allocated":835584,"data_stored":259296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30474240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064666+0000","last_change":"2023-09-30T00:39:23.063388+0000","last_active":"2023-09-30T00:39:23.064666+0000","last_peered":"2023-09-30T00:39:23.064666+0000","last_clean":"2023-09-30T00:39:23.064666+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:39:23.064666+0000","last_undegraded":"2023-09-30T00:39:23.064666+0000","last_fullsized":"2023-09-30T00:39:23.064666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:07.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064544+0000","last_change":"2023-09-30T00:39:23.063310+0000","last_active":"2023-09-30T00:39:23.064544+0000","last_peered":"2023-09-30T00:39:23.064544+0000","last_clean":"2023-09-30T00:39:23.064544+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:39:23.064544+0000","last_undegraded":"2023-09-30T00:39:23.064544+0000","last_fullsized":"2023-09-30T00:39:23.064544+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065449+0000","last_change":"2023-09-30T00:39:23.064396+0000","last_active":"2023-09-30T00:39:23.065449+0000","last_peered":"2023-09-30T00:39:23.065449+0000","last_clean":"2023-09-30T00:39:23.065449+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:39:23.065449+0000","last_undegraded":"2023-09-30T00:39:23.065449+0000","last_fullsized":"2023-09-30T00:39:23.065449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:12:59.211492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.899738+0000","last_change":"2023-09-30T00:39:23.899211+0000","last_active":"2023-09-30T00:39:23.899738+0000","last_peered":"2023-09-30T00:39:23.899738+0000","last_clean":"2023-09-30T00:39:23.899738+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:39:23.899738+0000","last_undegraded":"2023-09-30T00:39:23.899738+0000","last_fullsized":"2023-09-30T00:39:23.899738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:04:35.280729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064930+0000","last_change":"2023-09-30T00:39:23.062713+0000","last_active":"2023-09-30T00:39:23.064930+0000","last_peered":"2023-09-30T00:39:23.064930+0000","last_clean":"2023-09-30T00:39:23.064930+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:39:23.064930+0000","last_undegraded":"2023-09-30T00:39:23.064930+0000","last_fullsized":"2023-09-30T00:39:23.064930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:58.102938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065941+0000","last_change":"2023-09-30T00:39:23.065479+0000","last_active":"2023-09-30T00:39:23.065941+0000","last_peered":"2023-09-30T00:39:23.065941+0000","last_clean":"2023-09-30T00:39:23.065941+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:39:23.065941+0000","last_undegraded":"2023-09-30T00:39:23.065941+0000","last_fullsized":"2023-09-30T00:39:23.065941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:21:46.679489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065380+0000","last_change":"2023-09-30T00:39:23.064537+0000","last_active":"2023-09-30T00:39:23.065380+0000","last_peered":"2023-09-30T00:39:23.065380+0000","last_clean":"2023-09-30T00:39:23.065380+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:39:23.065380+0000","last_undegraded":"2023-09-30T00:39:23.065380+0000","last_fullsized":"2023-09-30T00:39:23.065380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:26:05.208826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065342+0000","last_change":"2023-09-30T00:39:23.064523+0000","last_active":"2023-09-30T00:39:23.065342+0000","last_peered":"2023-09-30T00:39:23.065342+0000","last_clean":"2023-09-30T00:39:23.065342+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:39:23.065342+0000","last_undegraded":"2023-09-30T00:39:23.065342+0000","last_fullsized":"2023-09-30T00:39:23.065342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:21:29.969487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.60699999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.499,"15min":0.497},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65800000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.71599999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63500000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.63800000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40799999999999997},"min":{"1min":0.29599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.56799999999999995,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.40500000000000003},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53400000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.435,"15min":0.442},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.505}]}]},{"osd":3,"up_from":12,"seq":51539607615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47699999999999998},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60399999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.39000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":1.046,"15min":1.046},"last":0.52300000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71099999999999997,"5min":0.747,"15min":0.747},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.73999999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.499,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.379},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53200000000000003,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.32700000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.436,"15min":0.434},"min":{"1min":0.313,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59499999999999997,"5min":0.625,"15min":0.625},"last":0.48799999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.221},{"interface":"front","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.56200000000000006,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.50700000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47399999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74399999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.748,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.48199999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.56299999999999994},"min":{"1min":0.40899999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70299999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.42299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.74299999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.45100000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.41699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.45200000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.379}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40400000000000003,"15min":0.432},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.46300000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.40200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.46899999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.44800000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74299999999999999,"5min":1.155,"15min":1.155},"last":0.54700000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.625,"15min":0.63300000000000001},"min":{"1min":0.47899999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.77200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.73099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.46200000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.69399999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56399999999999995}]}]},{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.57899999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.372,"5min":0.37,"15min":0.37},"max":{"1min":0.61199999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.497}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.372},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.375},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.621,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.57599999999999996}]}]},{"osd":0,"up_from":12,"seq":51539607615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.34899999999999998,"15min":0.33900000000000002},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.38500000000000001},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.54800000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38700000000000001,"15min":0.38100000000000001},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53300000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.624,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.501,"5min":0.437,"15min":0.437},"max":{"1min":0.78500000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.443},{"interface":"front","average":{"1min":0.624,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.5,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86599999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59899999999999998,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.61699999999999999},"min":{"1min":0.47399999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.376},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.624,"15min":0.621},"min":{"1min":0.56000000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:29.258 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:44:29.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:29.600 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:29.603 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:44:29.615 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T00:44:29.457539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30624,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":29760,"kb_avail":566200416,"statfs":{"total":579820584960,"available":579789225984,"internally_reserved":0,"allocated":835584,"data_stored":259296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30474240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064666+0000","last_change":"2023-09-30T00:39:23.063388+0000","last_active":"2023-09-30T00:39:23.064666+0000","last_peered":"2023-09-30T00:39:23.064666+0000","last_clean":"2023-09-30T00:39:23.064666+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:39:23.064666+0000","last_undegraded":"2023-09-30T00:39:23.064666+0000","last_fullsized":"2023-09-30T00:39:23.064666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:07.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064544+0000","last_change":"2023-09-30T00:39:23.063310+0000","last_active":"2023-09-30T00:39:23.064544+0000","last_peered":"2023-09-30T00:39:23.064544+0000","last_clean":"2023-09-30T00:39:23.064544+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:39:23.064544+0000","last_undegraded":"2023-09-30T00:39:23.064544+0000","last_fullsized":"2023-09-30T00:39:23.064544+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065449+0000","last_change":"2023-09-30T00:39:23.064396+0000","last_active":"2023-09-30T00:39:23.065449+0000","last_peered":"2023-09-30T00:39:23.065449+0000","last_clean":"2023-09-30T00:39:23.065449+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:39:23.065449+0000","last_undegraded":"2023-09-30T00:39:23.065449+0000","last_fullsized":"2023-09-30T00:39:23.065449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:12:59.211492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.899738+0000","last_change":"2023-09-30T00:39:23.899211+0000","last_active":"2023-09-30T00:39:23.899738+0000","last_peered":"2023-09-30T00:39:23.899738+0000","last_clean":"2023-09-30T00:39:23.899738+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:39:23.899738+0000","last_undegraded":"2023-09-30T00:39:23.899738+0000","last_fullsized":"2023-09-30T00:39:23.899738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:04:35.280729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064930+0000","last_change":"2023-09-30T00:39:23.062713+0000","last_active":"2023-09-30T00:39:23.064930+0000","last_peered":"2023-09-30T00:39:23.064930+0000","last_clean":"2023-09-30T00:39:23.064930+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:39:23.064930+0000","last_undegraded":"2023-09-30T00:39:23.064930+0000","last_fullsized":"2023-09-30T00:39:23.064930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:58.102938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065941+0000","last_change":"2023-09-30T00:39:23.065479+0000","last_active":"2023-09-30T00:39:23.065941+0000","last_peered":"2023-09-30T00:39:23.065941+0000","last_clean":"2023-09-30T00:39:23.065941+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:39:23.065941+0000","last_undegraded":"2023-09-30T00:39:23.065941+0000","last_fullsized":"2023-09-30T00:39:23.065941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:21:46.679489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065380+0000","last_change":"2023-09-30T00:39:23.064537+0000","last_active":"2023-09-30T00:39:23.065380+0000","last_peered":"2023-09-30T00:39:23.065380+0000","last_clean":"2023-09-30T00:39:23.065380+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:39:23.065380+0000","last_undegraded":"2023-09-30T00:39:23.065380+0000","last_fullsized":"2023-09-30T00:39:23.065380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:26:05.208826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065342+0000","last_change":"2023-09-30T00:39:23.064523+0000","last_active":"2023-09-30T00:39:23.065342+0000","last_peered":"2023-09-30T00:39:23.065342+0000","last_clean":"2023-09-30T00:39:23.065342+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:39:23.065342+0000","last_undegraded":"2023-09-30T00:39:23.065342+0000","last_fullsized":"2023-09-30T00:39:23.065342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:21:29.969487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.60699999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.499,"15min":0.497},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65800000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.71599999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63500000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.63800000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40799999999999997},"min":{"1min":0.29599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.56799999999999995,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.40500000000000003},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53400000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.435,"15min":0.442},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.505}]}]},{"osd":3,"up_from":12,"seq":51539607615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47699999999999998},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60399999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.39000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":1.046,"15min":1.046},"last":0.52300000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.67600000000000005}]},{"osd":2,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71099999999999997,"5min":0.747,"15min":0.747},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.73999999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.499,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.379},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53200000000000003,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.32700000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.436,"15min":0.434},"min":{"1min":0.313,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59499999999999997,"5min":0.625,"15min":0.625},"last":0.48799999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.221},{"interface":"front","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.56200000000000006,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.50700000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47399999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74399999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.748,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.48199999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.56299999999999994},"min":{"1min":0.40899999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70299999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.42299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.74299999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.45100000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.41699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.45200000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.379}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40400000000000003,"15min":0.432},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.46300000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.40200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.46899999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.44800000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74299999999999999,"5min":1.155,"15min":1.155},"last":0.54700000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.625,"15min":0.63300000000000001},"min":{"1min":0.47899999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.77200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.73099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.46200000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.69399999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.56399999999999995}]}]},{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.57899999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.372,"5min":0.37,"15min":0.37},"max":{"1min":0.61199999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.372},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.314},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.375},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.621,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.36499999999999999}]}]},{"osd":0,"up_from":12,"seq":51539607615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.34899999999999998,"15min":0.33900000000000002},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.38500000000000001},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.54800000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.27800000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38700000000000001,"15min":0.38100000000000001},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53300000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.624,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.501,"5min":0.437,"15min":0.437},"max":{"1min":0.78500000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.443},{"interface":"front","average":{"1min":0.624,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.5,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86599999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59899999999999998,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.61699999999999999},"min":{"1min":0.47399999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.376},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.624,"15min":0.621},"min":{"1min":0.56000000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:29.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:29.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:29.962 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"6147adfb-945f-4aac-97fa-4c6fe5df7b7a","created":"2023-09-30T00:39:10.895764+0000","modified":"2023-09-30T00:39:24.060378+0000","last_up_change":"2023-09-30T00:39:20.047161+0000","last_in_change":"2023-09-30T00:39:12.944606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-30T00:39:21.399773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"435ed398-a081-4f06-9104-8a7555961c2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":114351},{"type":"v1","addr":"172.21.15.29:6801","nonce":114351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":114351},{"type":"v1","addr":"172.21.15.29:6803","nonce":114351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":114351},{"type":"v1","addr":"172.21.15.29:6807","nonce":114351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":114351},{"type":"v1","addr":"172.21.15.29:6805","nonce":114351}]},"public_addr":"172.21.15.29:6801/114351","cluster_addr":"172.21.15.29:6803/114351","heartbeat_back_addr":"172.21.15.29:6807/114351","heartbeat_front_addr":"172.21.15.29:6805/114351","state":["exists","up"]},{"osd":1,"uuid":"4e245884-87c8-497e-af9e-a856480adf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":114350},{"type":"v1","addr":"172.21.15.29:6810","nonce":114350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":114350},{"type":"v1","addr":"172.21.15.29:6815","nonce":114350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":114350},{"type":"v1","addr":"172.21.15.29:6823","nonce":114350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":114350},{"type":"v1","addr":"172.21.15.29:6820","nonce":114350}]},"public_addr":"172.21.15.29:6810/114350","cluster_addr":"172.21.15.29:6815/114350","heartbeat_back_addr":"172.21.15.29:6823/114350","heartbeat_front_addr":"172.21.15.29:6820/114350","state":["exists","up"]},{"osd":2,"uuid":"7c620fed-a00f-40de-a548-64b3fbaf1f2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":114352},{"type":"v1","addr":"172.21.15.29:6811","nonce":114352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":114352},{"type":"v1","addr":"172.21.15.29:6814","nonce":114352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6819","nonce":114352},{"type":"v1","addr":"172.21.15.29:6821","nonce":114352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":114352},{"type":"v1","addr":"172.21.15.29:6817","nonce":114352}]},"public_addr":"172.21.15.29:6811/114352","cluster_addr":"172.21.15.29:6814/114352","heartbeat_back_addr":"172.21.15.29:6821/114352","heartbeat_front_addr":"172.21.15.29:6817/114352","state":["exists","up"]},{"osd":3,"uuid":"e0c974a6-9069-40ce-859f-4abcc75a5d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":142501},{"type":"v1","addr":"172.21.15.173:6817","nonce":142501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":142501},{"type":"v1","addr":"172.21.15.173:6819","nonce":142501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":142501},{"type":"v1","addr":"172.21.15.173:6823","nonce":142501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":142501},{"type":"v1","addr":"172.21.15.173:6821","nonce":142501}]},"public_addr":"172.21.15.173:6817/142501","cluster_addr":"172.21.15.173:6819/142501","heartbeat_back_addr":"172.21.15.173:6823/142501","heartbeat_front_addr":"172.21.15.173:6821/142501","state":["exists","up"]},{"osd":4,"uuid":"4956961b-3c7c-4268-a86b-ef77e63a3dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":142502},{"type":"v1","addr":"172.21.15.173:6809","nonce":142502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":142502},{"type":"v1","addr":"172.21.15.173:6811","nonce":142502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":142502},{"type":"v1","addr":"172.21.15.173:6815","nonce":142502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":142502},{"type":"v1","addr":"172.21.15.173:6813","nonce":142502}]},"public_addr":"172.21.15.173:6809/142502","cluster_addr":"172.21.15.173:6811/142502","heartbeat_back_addr":"172.21.15.173:6815/142502","heartbeat_front_addr":"172.21.15.173:6813/142502","state":["exists","up"]},{"osd":5,"uuid":"02c3c662-187e-4ed9-a0d1-45773d3f2b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":142503},{"type":"v1","addr":"172.21.15.173:6801","nonce":142503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":142503},{"type":"v1","addr":"172.21.15.173:6803","nonce":142503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":142503},{"type":"v1","addr":"172.21.15.173:6807","nonce":142503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":142503},{"type":"v1","addr":"172.21.15.173:6805","nonce":142503}]},"public_addr":"172.21.15.173:6801/142503","cluster_addr":"172.21.15.173:6803/142503","heartbeat_back_addr":"172.21.15.173:6807/142503","heartbeat_front_addr":"172.21.15.173:6805/142503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.108600+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.127036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.156519+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.003755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:18.016418+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:39:17.980520+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:30.978 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T00:44:30.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:44:31.143 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-09-30T00:44:31.143 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:44:31.144 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-09-30T00:44:31.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T00:44:31.505 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T00:44:31.517 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T00:44:31.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:44:31.674 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-09-30T00:44:31.674 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:44:31.674 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-09-30T00:44:31.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T00:44:32.038 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T00:44:32.050 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T00:44:32.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:44:32.206 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-09-30T00:44:32.207 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:44:32.207 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-09-30T00:44:32.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T00:44:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T00:44:32.584 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T00:44:32.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:44:32.742 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-09-30T00:44:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:44:32.742 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-09-30T00:44:32.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T00:44:33.104 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T00:44:33.116 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T00:44:33.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:44:33.275 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-09-30T00:44:33.275 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:44:33.275 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-09-30T00:44:33.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T00:44:33.638 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T00:44:33.650 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T00:44:33.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:44:33.807 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-09-30T00:44:33.808 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:44:33.808 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-09-30T00:44:33.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T00:44:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T00:44:34.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:34.527 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:44:34.541 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2023-09-30T00:44:33.458193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30624,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":29760,"kb_avail":566200416,"statfs":{"total":579820584960,"available":579789225984,"internally_reserved":0,"allocated":835584,"data_stored":259296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30474240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064666+0000","last_change":"2023-09-30T00:39:23.063388+0000","last_active":"2023-09-30T00:39:23.064666+0000","last_peered":"2023-09-30T00:39:23.064666+0000","last_clean":"2023-09-30T00:39:23.064666+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:39:23.064666+0000","last_undegraded":"2023-09-30T00:39:23.064666+0000","last_fullsized":"2023-09-30T00:39:23.064666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:07.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064544+0000","last_change":"2023-09-30T00:39:23.063310+0000","last_active":"2023-09-30T00:39:23.064544+0000","last_peered":"2023-09-30T00:39:23.064544+0000","last_clean":"2023-09-30T00:39:23.064544+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:39:23.064544+0000","last_undegraded":"2023-09-30T00:39:23.064544+0000","last_fullsized":"2023-09-30T00:39:23.064544+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:48:08.168118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065449+0000","last_change":"2023-09-30T00:39:23.064396+0000","last_active":"2023-09-30T00:39:23.065449+0000","last_peered":"2023-09-30T00:39:23.065449+0000","last_clean":"2023-09-30T00:39:23.065449+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:39:23.065449+0000","last_undegraded":"2023-09-30T00:39:23.065449+0000","last_fullsized":"2023-09-30T00:39:23.065449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:12:59.211492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.899738+0000","last_change":"2023-09-30T00:39:23.899211+0000","last_active":"2023-09-30T00:39:23.899738+0000","last_peered":"2023-09-30T00:39:23.899738+0000","last_clean":"2023-09-30T00:39:23.899738+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:39:23.899738+0000","last_undegraded":"2023-09-30T00:39:23.899738+0000","last_fullsized":"2023-09-30T00:39:23.899738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:04:35.280729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.064930+0000","last_change":"2023-09-30T00:39:23.062713+0000","last_active":"2023-09-30T00:39:23.064930+0000","last_peered":"2023-09-30T00:39:23.064930+0000","last_clean":"2023-09-30T00:39:23.064930+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:39:23.064930+0000","last_undegraded":"2023-09-30T00:39:23.064930+0000","last_fullsized":"2023-09-30T00:39:23.064930+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:40:58.102938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065941+0000","last_change":"2023-09-30T00:39:23.065479+0000","last_active":"2023-09-30T00:39:23.065941+0000","last_peered":"2023-09-30T00:39:23.065941+0000","last_clean":"2023-09-30T00:39:23.065941+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:39:23.065941+0000","last_undegraded":"2023-09-30T00:39:23.065941+0000","last_fullsized":"2023-09-30T00:39:23.065941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:21:46.679489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065380+0000","last_change":"2023-09-30T00:39:23.064537+0000","last_active":"2023-09-30T00:39:23.065380+0000","last_peered":"2023-09-30T00:39:23.065380+0000","last_clean":"2023-09-30T00:39:23.065380+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:39:23.065380+0000","last_undegraded":"2023-09-30T00:39:23.065380+0000","last_fullsized":"2023-09-30T00:39:23.065380+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:26:05.208826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:39:23.065342+0000","last_change":"2023-09-30T00:39:23.064523+0000","last_active":"2023-09-30T00:39:23.065342+0000","last_peered":"2023-09-30T00:39:23.065342+0000","last_clean":"2023-09-30T00:39:23.065342+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:39:23.065342+0000","last_undegraded":"2023-09-30T00:39:23.065342+0000","last_fullsized":"2023-09-30T00:39:23.065342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:39:22.051933+0000","last_clean_scrub_stamp":"2023-09-30T00:39:22.051933+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:21:29.969487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607617,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.60699999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.499,"15min":0.497},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46500000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.71599999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.51400000000000001,"15min":0.51800000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63500000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.502}]},{"osd":2,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68999999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.63800000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.40799999999999997},"min":{"1min":0.29599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.56799999999999995,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.40500000000000003},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35399999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.53400000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.435},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.435,"15min":0.442},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59699999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39300000000000002}]}]},{"osd":3,"up_from":12,"seq":51539607616,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47699999999999998},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60399999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.39000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":1.046,"15min":1.046},"last":0.66500000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.627},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71099999999999997,"5min":0.747,"15min":0.747},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.73999999999999999,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.316,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.499,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.39500000000000002,"15min":0.39100000000000001},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.53200000000000003,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:33 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.436,"15min":0.434},"min":{"1min":0.313,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59499999999999997,"5min":0.625,"15min":0.625},"last":0.55600000000000005}]}]},{"osd":2,"up_from":12,"seq":51539607616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.33100000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.56200000000000006,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33300000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.50700000000000001,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.378},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.47399999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.29199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55400000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74399999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70799999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.44500000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.748,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.48399999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56499999999999995,"15min":0.56299999999999994},"min":{"1min":0.40899999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70299999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.42299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.74299999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.503}]}]},{"osd":1,"up_from":12,"seq":51539607616,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.41699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39300000000000002,"15min":0.42399999999999999},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.45200000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40400000000000003,"15min":0.432},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.46300000000000002,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.40200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.46899999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59699999999999998,"15min":0.59099999999999997},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74299999999999999,"5min":1.155,"15min":1.155},"last":0.52900000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.625,"15min":0.63300000000000001},"min":{"1min":0.47899999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.77200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70999999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75800000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61399999999999999,"15min":0.61499999999999999},"min":{"1min":0.46200000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.69399999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.51700000000000002}]}]},{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.40699999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.59199999999999997,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.38,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.57899999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.62,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.49399999999999999,"15min":0.48899999999999999},"min":{"1min":0.372,"5min":0.37,"15min":0.37},"max":{"1min":0.61199999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.372},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.314},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.377,"15min":0.375},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61099999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.621,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.36499999999999999}]}]},{"osd":0,"up_from":12,"seq":51539607616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.34899999999999998,"15min":0.33900000000000002},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.315},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.38600000000000001,"15min":0.38500000000000001},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.54800000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.37}]},{"osd":2,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38700000000000001,"15min":0.38100000000000001},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53300000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41099999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.624,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.501,"5min":0.437,"15min":0.437},"max":{"1min":0.78500000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.496},{"interface":"front","average":{"1min":0.624,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.5,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86599999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59899999999999998,"15min":0.58999999999999997},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.61899999999999999,"15min":0.61699999999999999},"min":{"1min":0.47399999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.78800000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.54000000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:43:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.48199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.624,"15min":0.621},"min":{"1min":0.56000000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:34.546 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.546 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.546 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.546 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.546 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.546 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.547 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.547 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2023-09-30T00:39:22.051933+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=39, tm_sec=22, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=29, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:44:34.547 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T00:44:54.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-09-30T00:44:54.902 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-09-30T00:44:54.913 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2023-09-30T00:44:53.462546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":30624,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":29760,"kb_avail":566200416,"statfs":{"total":579820584960,"available":579789225984,"internally_reserved":0,"allocated":835584,"data_stored":259296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30474240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:32.382999+0000","last_change":"2023-09-30T00:44:32.382963+0000","last_active":"2023-09-30T00:44:32.382999+0000","last_peered":"2023-09-30T00:44:32.382999+0000","last_clean":"2023-09-30T00:44:32.382999+0000","last_became_active":"2023-09-30T00:39:23.063067+0000","last_became_peered":"2023-09-30T00:39:23.063067+0000","last_unstale":"2023-09-30T00:44:32.382999+0000","last_undegraded":"2023-09-30T00:44:32.382999+0000","last_fullsized":"2023-09-30T00:44:32.382999+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:32.382863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:32.382863+0000","last_clean_scrub_stamp":"2023-09-30T00:44:32.382863+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:30:41.001268+0000","scrub_duration":0.0019870069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:37.459188+0000","last_change":"2023-09-30T00:44:37.459157+0000","last_active":"2023-09-30T00:44:37.459188+0000","last_peered":"2023-09-30T00:44:37.459188+0000","last_clean":"2023-09-30T00:44:37.459188+0000","last_became_active":"2023-09-30T00:39:23.063002+0000","last_became_peered":"2023-09-30T00:39:23.063002+0000","last_unstale":"2023-09-30T00:44:37.459188+0000","last_undegraded":"2023-09-30T00:44:37.459188+0000","last_fullsized":"2023-09-30T00:44:37.459188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:37.459034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:37.459034+0000","last_clean_scrub_stamp":"2023-09-30T00:44:37.459034+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:50:06.527770+0000","scrub_duration":0.0016605579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:33.937021+0000","last_change":"2023-09-30T00:44:33.936985+0000","last_active":"2023-09-30T00:44:33.937021+0000","last_peered":"2023-09-30T00:44:33.937021+0000","last_clean":"2023-09-30T00:44:33.937021+0000","last_became_active":"2023-09-30T00:39:23.064058+0000","last_became_peered":"2023-09-30T00:39:23.064058+0000","last_unstale":"2023-09-30T00:44:33.937021+0000","last_undegraded":"2023-09-30T00:44:33.937021+0000","last_fullsized":"2023-09-30T00:44:33.937021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:33.936851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:33.936851+0000","last_clean_scrub_stamp":"2023-09-30T00:44:33.936851+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:37:38.849520+0000","scrub_duration":0.001870133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:35.407726+0000","last_change":"2023-09-30T00:44:35.407697+0000","last_active":"2023-09-30T00:44:35.407726+0000","last_peered":"2023-09-30T00:44:35.407726+0000","last_clean":"2023-09-30T00:44:35.407726+0000","last_became_active":"2023-09-30T00:39:23.898864+0000","last_became_peered":"2023-09-30T00:39:23.898864+0000","last_unstale":"2023-09-30T00:44:35.407726+0000","last_undegraded":"2023-09-30T00:44:35.407726+0000","last_fullsized":"2023-09-30T00:44:35.407726+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:35.407561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:35.407561+0000","last_clean_scrub_stamp":"2023-09-30T00:44:35.407561+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:31:33.387259+0000","scrub_duration":0.00180948,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:37.300130+0000","last_change":"2023-09-30T00:44:37.300101+0000","last_active":"2023-09-30T00:44:37.300130+0000","last_peered":"2023-09-30T00:44:37.300130+0000","last_clean":"2023-09-30T00:44:37.300130+0000","last_became_active":"2023-09-30T00:39:23.062365+0000","last_became_peered":"2023-09-30T00:39:23.062365+0000","last_unstale":"2023-09-30T00:44:37.300130+0000","last_undegraded":"2023-09-30T00:44:37.300130+0000","last_fullsized":"2023-09-30T00:44:37.300130+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:37.299991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:37.299991+0000","last_clean_scrub_stamp":"2023-09-30T00:44:37.299991+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:29:28.758358+0000","scrub_duration":0.0015511870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:36.790027+0000","last_change":"2023-09-30T00:44:36.789983+0000","last_active":"2023-09-30T00:44:36.790027+0000","last_peered":"2023-09-30T00:44:36.790027+0000","last_clean":"2023-09-30T00:44:36.790027+0000","last_became_active":"2023-09-30T00:39:23.065177+0000","last_became_peered":"2023-09-30T00:39:23.065177+0000","last_unstale":"2023-09-30T00:44:36.790027+0000","last_undegraded":"2023-09-30T00:44:36.790027+0000","last_fullsized":"2023-09-30T00:44:36.790027+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:36.789873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:36.789873+0000","last_clean_scrub_stamp":"2023-09-30T00:44:36.789873+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:19:36.571273+0000","scrub_duration":0.0021647540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:36.047044+0000","last_change":"2023-09-30T00:44:36.047015+0000","last_active":"2023-09-30T00:44:36.047044+0000","last_peered":"2023-09-30T00:44:36.047044+0000","last_clean":"2023-09-30T00:44:36.047044+0000","last_became_active":"2023-09-30T00:39:23.064164+0000","last_became_peered":"2023-09-30T00:39:23.064164+0000","last_unstale":"2023-09-30T00:44:36.047044+0000","last_undegraded":"2023-09-30T00:44:36.047044+0000","last_fullsized":"2023-09-30T00:44:36.047044+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:36.046915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:36.046915+0000","last_clean_scrub_stamp":"2023-09-30T00:44:36.046915+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:24:53.943391+0000","scrub_duration":0.0018699369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:42.978812+0000","last_change":"2023-09-30T00:44:42.978764+0000","last_active":"2023-09-30T00:44:42.978812+0000","last_peered":"2023-09-30T00:44:42.978812+0000","last_clean":"2023-09-30T00:44:42.978812+0000","last_became_active":"2023-09-30T00:39:23.064147+0000","last_became_peered":"2023-09-30T00:39:23.064147+0000","last_unstale":"2023-09-30T00:44:42.978812+0000","last_undegraded":"2023-09-30T00:44:42.978812+0000","last_fullsized":"2023-09-30T00:44:42.978812+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:42.978649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:42.978649+0000","last_clean_scrub_stamp":"2023-09-30T00:44:42.978649+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:58:02.905337+0000","scrub_duration":0.0020033339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607621,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51100000000000001,"15min":0.52900000000000003},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70699999999999996,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.495,"15min":0.496},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65800000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64700000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51000000000000001,"15min":0.52000000000000002},"min":{"1min":0.40100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.64500000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68999999999999995,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52200000000000002,"15min":0.53700000000000003},"min":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53300000000000003,"15min":0.54300000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.64800000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.623}]},{"osd":3,"last update":"Sat Sep 30 00:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40899999999999997,"15min":0.40699999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.38800000000000001,"15min":0.40200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.49299999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48799999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41199999999999998,"15min":0.42299999999999999},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65900000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42599999999999999,"15min":0.439},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.51900000000000002,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.53500000000000003}]}]},{"osd":3,"up_from":12,"seq":51539607620,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.47599999999999998},"min":{"1min":0.34399999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48799999999999999,"15min":0.497},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66500000000000004,"5min":1.046,"15min":1.046},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.624},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71699999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.40100000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68700000000000006,"5min":0.747,"15min":0.747},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56200000000000006,"15min":0.55900000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70299999999999996,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38100000000000001,"15min":0.376},"min":{"1min":0.251,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48599999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.50600000000000001,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.32000000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.30199999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.53400000000000003,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42799999999999999,"15min":0.432},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.55600000000000005,"5min":0.625,"15min":0.625},"last":0.36599999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.495,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.307},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.37,"15min":0.36599999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59399999999999997,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.36799999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.36699999999999999,"15min":0.35899999999999999},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.29099999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.34100000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76100000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57899999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73899999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56699999999999995,"15min":0.56399999999999995},"min":{"1min":0.43099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.505},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56399999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607620,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.36299999999999999},"min":{"1min":0.251,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.499,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.373,"15min":0.41699999999999998},"min":{"1min":0.25900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.46999999999999997,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.44700000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.38500000000000001,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.44800000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.38800000000000001,"15min":0.40000000000000002},"min":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.49399999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.753,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59299999999999997,"15min":0.61699999999999999},"min":{"1min":0.48099999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.71499999999999997,"5min":1.155,"15min":1.155},"last":0.70199999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":0.629},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.78900000000000003,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73099999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60399999999999998,"15min":0.60699999999999998},"min":{"1min":0.46700000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.83299999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.46300000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78200000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.73399999999999999}]}]},{"osd":5,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.496,"15min":0.505},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.316,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45400000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.55700000000000005},"min":{"1min":0.41199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.63600000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.505,"15min":0.49299999999999999},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.379,"15min":0.372},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.309},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.375,"15min":0.375},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.50700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38500000000000001,"15min":0.38400000000000001},"min":{"1min":0.28299999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54500000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60799999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.35399999999999998}]}]},{"osd":0,"up_from":12,"seq":51539607620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":43216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.34699999999999998,"15min":0.33800000000000002},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.49399999999999999,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38200000000000001,"15min":0.38400000000000001},"min":{"1min":0.27800000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.47899999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38500000000000001,"15min":0.38100000000000001},"min":{"1min":0.29199999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51100000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.441},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.54000000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.70299999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.60099999999999998,"15min":0.57899999999999996},"min":{"1min":0.42899999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.77500000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.66600000000000004}]},{"osd":4,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.59099999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76400000000000001,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.74099999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60599999999999998,"15min":0.59299999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80800000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.625,"15min":0.621},"min":{"1min":0.45200000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:54.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:44:55.276 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:44:55.278 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:44:55.278 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:44:55.278 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:44:55.278 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.312 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:44:55.313 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:44:55.313 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.376 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:44:55.376 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:44:55.377 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.446 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:44:55.446 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:44:55.446 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.483 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:44:55.483 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T00:44:55.483 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.553 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T00:44:55.554 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T00:44:55.554 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.619 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T00:44:55.620 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:44:55.620 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:44:55.620 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.648 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:44:55.648 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:44:55.648 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:44:55.648 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:44:55.701 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:44:55.701 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:44:55.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-09-30T00:44:55.779 INFO:teuthology.orchestra.run.smithi029.stdout:1696034357.42288 mgr.x (mgr.4102) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:44:55.779 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:44:55.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-09-30T00:44:55.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2023-09-30T00:44:55.936 INFO:teuthology.orchestra.run.smithi029.stdout:1696034357.42288 mgr.x (mgr.4102) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:44:55.937 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:44:55.937 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:44:56.227 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:44:56.227 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:44:56.351 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:44:56.351 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:44:56.430 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:44:56.430 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:44:56.528 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:44:56.528 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T00:44:56.595 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:44:56.595 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T00:44:56.661 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:44:56.662 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/data/mon.a.tgz 2023-09-30T00:44:56.662 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-09-30T00:44:56.686 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.UoqCCYwhI9 2023-09-30T00:44:56.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.UoqCCYwhI9 2023-09-30T00:44:56.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.UoqCCYwhI9 2023-09-30T00:44:56.924 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.UoqCCYwhI9 is 168KB 2023-09-30T00:44:56.977 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.UoqCCYwhI9 2023-09-30T00:44:57.000 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:44:57.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:44:57.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:44:57.075 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:44:57.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:44:57.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:44:57.150 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:44:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:44:57.160 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:44:57.162 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/remote/smithi029/crash 2023-09-30T00:44:57.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:44:57.238 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/remote/smithi173/crash 2023-09-30T00:44:57.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:44:57.278 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:44:57.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:44:57.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:44:57.828 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:44:57.828 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/remote/smithi029/log 2023-09-30T00:44:57.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:44:58.032 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/remote/smithi173/log 2023-09-30T00:44:58.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:44:58.096 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:44:58.160 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=97f30460e09745b489f2b25199ec9179 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:44:58.162 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:44:58.185 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:44:58.194 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:44:58.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:44:58.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:44:58.289 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:44:58.298 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:44:58.298 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:44:58.301 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:44:58.318 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2023-09-30T00:44:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:bash: ntpq: command not found 2023-09-30T00:44:58.322 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:44:58.322 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2023-09-30T00:44:58.323 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -328us[ -348us] +/- 82ms 2023-09-30T00:44:58.323 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -361us[ -381us] +/- 47ms 2023-09-30T00:44:58.323 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +834us[ +813us] +/- 67ms 2023-09-30T00:44:58.323 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:44:58.326 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:44:58.326 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-09-30T00:44:58.326 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 -339us[ -366us] +/- 82ms 2023-09-30T00:44:58.326 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -386us[ -414us] +/- 47ms 2023-09-30T00:44:58.327 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +807us[ +779us] +/- 67ms 2023-09-30T00:44:58.327 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:44:58.327 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:44:58.336 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:44:58.336 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:44:58.345 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:44:58.476 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:44:58.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:44:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:44:58.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:44:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:44:58.634 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:44:58.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:44:58.700 INFO:teuthology.task.internal:Duration was 1126.939945 seconds 2023-09-30T00:44:58.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:44:58.709 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:44:58.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:44:58.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:44:58.759 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:44:58.769 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:44:59.003 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:44:59.004 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-09-30T00:44:59.004 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:44:59.033 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2023-09-30T00:44:59.033 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:44:59.056 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:44:59.057 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:44:59.077 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:44:59.135 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:44:59.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:44:59.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:44:59.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:44:59.250 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:44:59.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:44:59.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:44:59.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:44:59.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:44:59.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:44:59.358 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-09-30T00:44:59.378 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2023-09-30T00:44:59.408 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:44:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:44:59.448 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:44:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:44:59.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:44:59.475 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:44:59.476 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/remote/smithi029 2023-09-30T00:44:59.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:44:59.534 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704/remote/smithi173 2023-09-30T00:44:59.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:44:59.571 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:44:59.571 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:44:59.574 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:44:59.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:44:59.634 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:44:59.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:44:59.643 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:44:59.643 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:44:59.646 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:44:59.666 INFO:teuthology.orchestra.run.smithi029.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:44 /home/ubuntu/cephtest 2023-09-30T00:44:59.681 INFO:teuthology.orchestra.run.smithi173.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:44 /home/ubuntu/cephtest 2023-09-30T00:44:59.682 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:44:59.692 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:44:59.711 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:44:59.736 DEBUG:teuthology.nuke:shortname: smithi029 2023-09-30T00:44:59.737 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:44:59.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:05:11.697690', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsY3dkdayRkNgoX0Qlg5cj5pGUZND1jWoYlelzq7aQsmzcxZPmCnUaXm9pGTailhMqhJwBVeBbAoLWLY3KOQhs='} 2023-09-30T00:44:59.764 DEBUG:teuthology.nuke:shortname: smithi173 2023-09-30T00:44:59.764 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:44:59.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406704', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:05:11.695719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrRO3VWph3/GHDDdUoVzBsEbNzIV+12yJdhkmN9+h88n62+fdyd3eNP3rnljTbH85R5E4rDMHojjQVNTbhUoTk='} 2023-09-30T00:44:59.824 INFO:teuthology.orchestra.console.smithi029:Power off 2023-09-30T00:44:59.825 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:44:59.853 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-09-30T00:44:59.853 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:44:59.867 INFO:teuthology.orchestra.console.smithi173:Power off 2023-09-30T00:44:59.868 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:44:59.877 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-09-30T00:44:59.890 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2023-09-30T00:44:59.890 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:44:59.912 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2023-09-30T00:45:03.879 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:45:03.914 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:45:03.990 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-09-30T00:45:04.023 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2023-09-30T00:45:07.995 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:45:08.025 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:45:08.106 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-09-30T00:45:08.134 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2023-09-30T00:45:08.207 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-09-30T00:45:08.235 INFO:teuthology.orchestra.console.smithi173:Power off completed 2023-09-30T00:45:08.340 INFO:teuthology.run:Summary data: description: rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} duration: 1126.9399449825287 failure_reason: '"1696034357.42288 mgr.x (mgr.4102) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=97f30460e09745b489f2b25199ec9179 status: fail success: false 2023-09-30T00:45:08.340 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:45:08.425 INFO:teuthology.run:FAIL