2023-12-01T21:44:14.246 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-01T21:44:14.246 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306 --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} --owner scheduled_yuriw@teuthology --name yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/orig.config.yaml 2023-12-01T21:44:14.281 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T21:44:14.365 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306 branch: wip-yuri8-testing-2023-11-29-0706-pacific description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} email: null first_in_suite: false job_id: '7474306' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri8-testing-2023-11-29-0706-pacific ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-11-29-0706-pacific sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 8528 sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2023-11-29-0706-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHXlX9zAwNcMwQQU3DB/EAeYCGpSjW2Nux9nhT2vXGdTFPf5SeLso4wUcOT9/x/mg0OnwCREdKDZcPGAiTVcXiY= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWQdVma+lyjNWkZyjNVt3uVkytJ8p1wyCBD+94MlywamkEMtBb61G4EUuRHQxCXZHEFEPB8KciAtaYAKz6p/mk= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-01_16:53:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-01T21:44:14.366 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa; will attempt to use it 2023-12-01T21:44:14.366 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks 2023-12-01T21:44:14.366 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-01T21:44:14.368 INFO:teuthology.task.internal:Checking packages... 2023-12-01T21:44:14.392 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9' 2023-12-01T21:44:14.392 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-01T21:44:14.392 INFO:teuthology.packaging:ref: None 2023-12-01T21:44:14.392 INFO:teuthology.packaging:tag: None 2023-12-01T21:44:14.392 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-11-29-0706-pacific 2023-12-01T21:44:14.392 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T21:44:14.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri8-testing-2023-11-29-0706-pacific 2023-12-01T21:44:14.666 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-328.g0dc074ce 2023-12-01T21:44:14.666 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-01T21:44:14.673 INFO:teuthology.task.internal:no buildpackages task found 2023-12-01T21:44:14.673 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-01T21:44:14.696 INFO:teuthology.task.internal:Saving configuration 2023-12-01T21:44:14.711 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-01T21:44:14.717 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T21:44:14.742 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:38:31.002543', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHXlX9zAwNcMwQQU3DB/EAeYCGpSjW2Nux9nhT2vXGdTFPf5SeLso4wUcOT9/x/mg0OnwCREdKDZcPGAiTVcXiY='} 2023-12-01T21:44:14.767 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:38:31.003982', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWQdVma+lyjNWkZyjNVt3uVkytJ8p1wyCBD+94MlywamkEMtBb61G4EUuRHQxCXZHEFEPB8KciAtaYAKz6p/mk='} 2023-12-01T21:44:14.767 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-01T21:44:14.774 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-12-01T21:44:14.774 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-12-01T21:44:14.774 INFO:teuthology.run_tasks:Running task console_log... 2023-12-01T21:44:14.842 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0e217f5ee0>, signals=[15]) 2023-12-01T21:44:14.843 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-01T21:44:14.849 INFO:teuthology.task.internal:Opening connections... 2023-12-01T21:44:14.849 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T21:44:14.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:44:14.923 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T21:44:14.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:44:14.992 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-01T21:44:14.999 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2023-12-01T21:44:15.017 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2023-12-01T21:44:15.017 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2023-12-01T21:44:15.074 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Red Hat Enterprise Linux" 2023-12-01T21:44:15.074 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8.4 (Ootpa)" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:ID="rhel" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="fedora" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8.4" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.redhat.com/" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-01T21:44:15.075 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-01T21:44:15.076 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:44:15.076 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-01T21:44:15.076 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-12-01T21:44:15.076 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-01T21:44:15.076 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-12-01T21:44:15.076 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2023-12-01T21:44:15.102 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-12-01T21:44:15.119 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-12-01T21:44:15.120 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-12-01T21:44:15.176 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Red Hat Enterprise Linux" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8.4 (Ootpa)" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:ID="rhel" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="fedora" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8.4" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.redhat.com/" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-01T21:44:15.177 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-01T21:44:15.178 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:44:15.178 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-01T21:44:15.179 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-12-01T21:44:15.179 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-01T21:44:15.179 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-12-01T21:44:15.179 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-12-01T21:44:15.201 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-01T21:44:15.212 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-01T21:44:15.222 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-01T21:44:15.223 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2023-12-01T21:44:15.226 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-12-01T21:44:15.241 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-01T21:44:15.247 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-01T21:44:15.247 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2023-12-01T21:44:15.282 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-12-01T21:44:15.297 INFO:teuthology.orchestra.run.smithi039.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-12-01T21:44:15.299 INFO:teuthology.orchestra.run.smithi170.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-12-01T21:44:15.299 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-01T21:44:15.354 INFO:teuthology.run_tasks:Running task kernel... 2023-12-01T21:44:15.368 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-01T21:44:15.368 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-01T21:44:15.369 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-01T21:44:15.369 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-01T21:44:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:44:15.385 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-12-01T21:44:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:44:15.386 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-12-01T21:44:15.441 INFO:teuthology.orchestra.run.smithi170.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-12-01T21:44:15.442 INFO:teuthology.task.kernel:Running kernel on smithi170: 4.18.0-305.25.1.el8_4.x86_64 2023-12-01T21:44:15.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2023-12-01T21:44:15.443 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-12-01T21:44:15.443 INFO:teuthology.task.kernel:Running kernel on smithi039: 4.18.0-305.25.1.el8_4.x86_64 2023-12-01T21:44:15.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2023-12-01T21:44:16.808 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:44:16.809 INFO:teuthology.orchestra.run.smithi170.stdout:Unable to read consumer identity 2023-12-01T21:44:16.825 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:44:16.826 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-12-01T21:44:18.228 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-12-01T21:44:18.251 INFO:teuthology.orchestra.run.smithi170.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-12-01T21:44:19.663 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-12-01T21:44:19.679 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-12-01T21:44:22.914 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-12-01T21:44:23.191 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-12-01T21:44:23.652 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-12-01T21:44:23.652 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-12-01T21:44:23.652 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-12-01T21:44:23.813 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:44:23.814 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:44:23.814 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:44:23.932 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-12-01T21:44:23.932 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-12-01T21:44:23.932 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-12-01T21:44:23.935 DEBUG:teuthology.orchestra.run.smithi170:> echo no | sudo yum reinstall kernel || true 2023-12-01T21:44:24.100 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:44:24.101 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:44:24.101 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:44:24.217 DEBUG:teuthology.orchestra.run.smithi039:> echo no | sudo yum reinstall kernel || true 2023-12-01T21:44:24.318 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:44:24.319 INFO:teuthology.orchestra.run.smithi170.stdout:Unable to read consumer identity 2023-12-01T21:44:24.584 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:44:24.584 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-12-01T21:44:24.611 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:02 ago on Fri 01 Dec 2023 09:44:22 PM UTC. 2023-12-01T21:44:24.746 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-01T21:44:24.747 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:24.747 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:24.783 INFO:teuthology.orchestra.run.smithi170.stderr:Error: No packages marked for reinstall. 2023-12-01T21:44:24.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum reinstall -y kernel || true 2023-12-01T21:44:24.894 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:01 ago on Fri 01 Dec 2023 09:44:23 PM UTC. 2023-12-01T21:44:25.040 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-01T21:44:25.040 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:25.040 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:25.061 INFO:teuthology.orchestra.run.smithi039.stderr:Error: No packages marked for reinstall. 2023-12-01T21:44:25.112 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum reinstall -y kernel || true 2023-12-01T21:44:25.184 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:44:25.184 INFO:teuthology.orchestra.run.smithi170.stdout:Unable to read consumer identity 2023-12-01T21:44:25.476 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:03 ago on Fri 01 Dec 2023 09:44:22 PM UTC. 2023-12-01T21:44:25.478 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:44:25.478 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-12-01T21:44:25.611 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-01T21:44:25.611 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:25.612 INFO:teuthology.orchestra.run.smithi170.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:25.632 INFO:teuthology.orchestra.run.smithi170.stderr:Error: No packages marked for reinstall. 2023-12-01T21:44:25.690 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:44:25.778 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:02 ago on Fri 01 Dec 2023 09:44:23 PM UTC. 2023-12-01T21:44:25.815 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:25.815 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:25.815 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:25.816 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-01T21:44:25.816 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-01T21:44:25.816 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-01T21:44:25.816 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2023-12-01T21:44:25.816 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:44:25.914 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-12-01T21:44:25.914 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:25.914 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-01T21:44:25.925 INFO:teuthology.orchestra.run.smithi039.stderr:Error: No packages marked for reinstall. 2023-12-01T21:44:25.941 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:25.941 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:25.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.02 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 99.el8_4.1 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Group : System Environment/Base 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 9264631 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2023-12-01T21:44:25.968 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Relocations : (not relocatable) 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : Red Hat, Inc. 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : Red Hat, Inc. 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-01T21:44:25.969 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2023-12-01T21:44:25.970 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:44:25.970 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2023-12-01T21:44:25.970 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-01T21:44:25.970 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:25.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-01T21:44:25.979 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:44:26.110 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:26.111 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:26.111 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:26.111 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-01T21:44:26.111 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-01T21:44:26.111 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-01T21:44:26.111 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2023-12-01T21:44:26.111 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T21:44:26.241 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:26.241 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:26.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.02 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 99.el8_4.1 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Group : System Environment/Base 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 9264631 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2023-12-01T21:44:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Relocations : (not relocatable) 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : Red Hat, Inc. 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : Red Hat, Inc. 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2023-12-01T21:44:26.271 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:44:26.272 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2023-12-01T21:44:26.272 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-01T21:44:26.273 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:26.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-01T21:44:27.329 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-12-01T21:44:27.466 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2023-12-01T21:44:29.097 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-12-01T21:44:29.099 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-12-01T21:44:29.113 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.VJoHcZy6Yy 2023-12-01T21:44:29.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VJoHcZy6Yy 2023-12-01T21:44:29.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.VJoHcZy6Yy 2023-12-01T21:44:29.298 INFO:teuthology.orchestra.run.smithi039.stderr:done 2023-12-01T21:44:29.299 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-12-01T21:44:29.313 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.AVCEr3jHjP 2023-12-01T21:44:29.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AVCEr3jHjP 2023-12-01T21:44:29.343 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.VJoHcZy6Yy is 6KB 2023-12-01T21:44:29.385 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.AVCEr3jHjP 2023-12-01T21:44:29.392 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.VJoHcZy6Yy 2023-12-01T21:44:29.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2023-12-01T21:44:29.473 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-12-01T21:44:29.473 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-12-01T21:44:29.473 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-12-01T21:44:29.473 INFO:teuthology.orchestra.run.smithi170.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-12-01T21:44:29.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:29.545 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.AVCEr3jHjP is 6KB 2023-12-01T21:44:29.594 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.AVCEr3jHjP 2023-12-01T21:44:29.608 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2023-12-01T21:44:29.648 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-12-01T21:44:29.688 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-12-01T21:44:29.688 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-12-01T21:44:29.688 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-12-01T21:44:29.688 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-12-01T21:44:29.690 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-12-01T21:44:29.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2023-12-01T21:44:59.680 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-01T21:44:59.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:44:59.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:44:59.856 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-01T21:44:59.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:44:59.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:45:18.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-12-01T21:45:18.183 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-12-01T21:45:27.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:45:27.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:45:27.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:45:27.196 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:45:30.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-12-01T21:45:30.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-12-01T21:45:42.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:45:42.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:45:42.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:45:42.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:45:45.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-12-01T21:45:45.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-12-01T21:46:00.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:46:00.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:46:00.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:46:00.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:46:03.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-12-01T21:46:21.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:46:21.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:46:21.853 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-12-01T21:46:22.105 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:46:22.105 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-12-01T21:46:22.106 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-12-01T21:46:22.121 INFO:teuthology.orchestra.run.smithi170.stdout:4.18.0-425.3.1.el8.x86_64 2023-12-01T21:46:22.121 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:46:22.121 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-01T21:46:22.121 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-12-01T21:46:23.122 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-01T21:46:23.123 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-01T21:46:23.211 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2023-12-01T21:46:23.227 DEBUG:teuthology.parallel:result is None 2023-12-01T21:47:00.365 DEBUG:teuthology.orchestra.remote:timed out 2023-12-01T21:47:18.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:47:18.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:47:18.637 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-12-01T21:47:18.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:47:18.892 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-12-01T21:47:18.893 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-12-01T21:47:18.909 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-425.3.1.el8.x86_64 2023-12-01T21:47:18.909 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-12-01T21:47:18.910 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-01T21:47:18.910 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-12-01T21:47:19.911 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-01T21:47:19.911 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-01T21:47:20.009 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2023-12-01T21:47:20.022 DEBUG:teuthology.parallel:result is None 2023-12-01T21:47:20.022 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-01T21:47:20.029 INFO:teuthology.task.internal:Creating test directory... 2023-12-01T21:47:20.029 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-01T21:47:20.031 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-01T21:47:20.048 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-01T21:47:20.064 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-01T21:47:20.077 INFO:teuthology.task.internal:Creating archive directory... 2023-12-01T21:47:20.077 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-01T21:47:20.089 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-01T21:47:20.124 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-01T21:47:20.135 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-01T21:47:20.136 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-01T21:47:20.160 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-01T21:47:20.197 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:47:20.207 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:47:20.208 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:47:20.219 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T21:47:20.221 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-01T21:47:20.232 INFO:teuthology.task.internal:Configuring sudo... 2023-12-01T21:47:20.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-01T21:47:20.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-01T21:47:20.276 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-01T21:47:20.287 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-01T21:47:20.287 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-01T21:47:20.303 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-01T21:47:20.332 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:47:20.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:47:20.455 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:47:20.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:47:20.579 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T21:47:20.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-01T21:47:20.645 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:47:20.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T21:47:20.730 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:47:20.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T21:47:20.853 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T21:47:20.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-01T21:47:20.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2023-12-01T21:47:20.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-12-01T21:47:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T21:47:21.016 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T21:47:21.450 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-01T21:47:21.459 INFO:teuthology.task.internal:Starting timer... 2023-12-01T21:47:21.459 INFO:teuthology.run_tasks:Running task pcp... 2023-12-01T21:47:21.484 INFO:teuthology.run_tasks:Running task selinux... 2023-12-01T21:47:21.496 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-01T21:47:21.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2023-12-01T21:47:21.550 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: [ OK ] 2023-12-01T21:47:21.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2023-12-01T21:47:21.608 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: [ OK ] 2023-12-01T21:47:21.610 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-01T21:47:21.610 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2023-12-01T21:47:21.631 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2023-12-01T21:47:21.631 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2023-12-01T21:47:21.654 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2023-12-01T21:47:21.654 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2023-12-01T21:47:21.654 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T21:47:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:47:21.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T21:47:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T21:47:21.724 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-01T21:47:21.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2023-12-01T21:47:21.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2023-12-01T21:47:21.790 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-01T21:47:21.803 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-01T21:47:21.809 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-01T21:47:21.920 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-01T21:47:21.941 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-01T21:47:21.942 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-01T21:55:49.192 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2023-12-01T21:55:49.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:55:49.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:55:49.267 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-12-01T21:55:49.338 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-12-01T21:55:49.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:55:49.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T21:55:49.409 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-12-01T21:55:49.432 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-01T21:55:49.433 INFO:teuthology.run_tasks:Running task clock... 2023-12-01T21:55:49.448 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-01T21:55:49.448 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-01T21:55:49.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T21:55:49.451 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-01T21:55:49.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T21:55:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-01T21:55:49.497 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-01T21:55:49.504 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-01T21:55:49.518 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-01T21:55:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2023-12-01T21:55:49.542 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2023-12-01T21:55:49.551 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2023-12-01T21:55:49.556 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-01T21:55:49.561 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2023-12-01T21:55:49.570 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-01T21:55:49.575 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-01T21:55:49.588 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-01T21:55:49.638 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-12-01T21:55:49.639 INFO:teuthology.orchestra.run.smithi039.stdout:210 Number of sources = 4 2023-12-01T21:55:49.640 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T21:55:49.640 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-12-01T21:55:49.640 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.640 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.640 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.640 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.653 INFO:teuthology.orchestra.run.smithi170.stderr:bash: ntpq: command not found 2023-12-01T21:55:49.655 INFO:teuthology.orchestra.run.smithi170.stdout:210 Number of sources = 4 2023-12-01T21:55:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T21:55:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-12-01T21:55:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.656 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T21:55:49.656 INFO:teuthology.run_tasks:Running task install... 2023-12-01T21:55:49.665 DEBUG:teuthology.task.install:project ceph 2023-12-01T21:55:49.665 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'}} 2023-12-01T21:55:49.665 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'} 2023-12-01T21:55:49.666 INFO:teuthology.task.install:Using flavor: default 2023-12-01T21:55:49.671 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-01T21:55:49.671 INFO:teuthology.task.install:extra packages: [] 2023-12-01T21:55:49.672 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'tag': None, 'wait_for_package': False} 2023-12-01T21:55:49.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T21:55:49.674 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'tag': None, 'wait_for_package': False} 2023-12-01T21:55:49.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T21:55:49.896 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/ 2023-12-01T21:55:49.897 INFO:teuthology.task.install.rpm:Package version is 16.2.14-328.g0dc074ce 2023-12-01T21:55:49.901 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/ 2023-12-01T21:55:49.901 INFO:teuthology.task.install.rpm:Package version is 16.2.14-328.g0dc074ce 2023-12-01T21:55:50.072 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-01T21:55:50.072 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T21:55:50.072 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-01T21:55:50.074 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-11-29-0706-pacific/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-01T21:55:50.074 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T21:55:50.075 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-01T21:55:50.099 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-01T21:55:50.100 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-01T21:55:50.103 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-01T21:55:50.103 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-01T21:55:50.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-01T21:55:50.170 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-01T21:55:50.240 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-01T21:55:50.248 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-01T21:55:50.303 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2023-12-01T21:55:50.305 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2023-12-01T21:55:50.313 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2023-12-01T21:55:50.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2023-12-01T21:55:50.696 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:55:50.726 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:55:51.018 INFO:teuthology.orchestra.run.smithi039.stdout:44 files removed 2023-12-01T21:55:51.023 INFO:teuthology.orchestra.run.smithi170.stdout:44 files removed 2023-12-01T21:55:51.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2023-12-01T21:55:51.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2023-12-01T21:55:51.423 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:55:51.445 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:55:52.299 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 173 kB/s | 76 kB 00:00 2023-12-01T21:55:52.300 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 185 kB/s | 76 kB 00:00 2023-12-01T21:55:52.676 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 46 kB/s | 15 kB 00:00 2023-12-01T21:55:52.881 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 28 kB/s | 15 kB 00:00 2023-12-01T21:55:53.229 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 2.8 kB/s | 1.4 kB 00:00 2023-12-01T21:55:53.340 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 43 kB/s | 3.5 kB 00:00 2023-12-01T21:55:53.562 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 2.2 kB/s | 1.4 kB 00:00 2023-12-01T21:55:53.672 INFO:teuthology.orchestra.run.smithi170.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-12-01T21:55:54.900 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-12-01T21:55:55.220 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2023-12-01T21:55:58.263 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-12-01T21:55:58.647 INFO:teuthology.orchestra.run.smithi170.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 126 MB/s | 38 MB 00:00 2023-12-01T21:55:58.784 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-12-01T21:55:59.186 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 105 MB/s | 33 MB 00:00 2023-12-01T21:56:04.219 INFO:teuthology.orchestra.run.smithi170.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 MB/s | 33 MB 00:00 2023-12-01T21:56:04.620 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 MB/s | 38 MB 00:00 2023-12-01T21:56:12.604 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:56:12.607 INFO:teuthology.orchestra.run.smithi170.stdout:==================================================================================================== 2023-12-01T21:56:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:56:12.607 INFO:teuthology.orchestra.run.smithi170.stdout:==================================================================================================== 2023-12-01T21:56:12.607 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:56:12.607 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 12 M 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 3.7 M 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 4.2 M 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 6.1 M 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 24 M 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 26 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 825 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 493 k 2023-12-01T21:56:12.608 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 3.8 M 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 45 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 99 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 215 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 390 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 369 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 114 k 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout:==================================================================================================== 2023-12-01T21:56:12.609 INFO:teuthology.orchestra.run.smithi170.stdout:Install 21 Packages 2023-12-01T21:56:12.610 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2023-12-01T21:56:12.610 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:12.610 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 57 M 2023-12-01T21:56:12.610 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:56:14.019 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout:==================================================================================================== 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout:==================================================================================================== 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 12 M 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 3.7 M 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 4.2 M 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 6.1 M 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 24 M 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 26 k 2023-12-01T21:56:14.022 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 825 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 493 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 3.8 M 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 45 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 99 k 2023-12-01T21:56:14.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 215 k 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 390 k 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 369 k 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 114 k 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout:==================================================================================================== 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout:Install 21 Packages 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:56:14.024 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 57 M 2023-12-01T21:56:14.025 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:56:14.286 INFO:teuthology.orchestra.run.smithi170.stdout:(1/23): ceph-base-16.2.14-328.g0dc074ce.el8.x86 4.6 MB/s | 6.1 MB 00:01 2023-12-01T21:56:14.365 INFO:teuthology.orchestra.run.smithi170.stdout:(2/23): ceph-selinux-16.2.14-328.g0dc074ce.el8. 326 kB/s | 26 kB 00:00 2023-12-01T21:56:14.513 INFO:teuthology.orchestra.run.smithi170.stdout:(3/23): libcephfs2-16.2.14-328.g0dc074ce.el8.x8 5.5 MB/s | 825 kB 00:00 2023-12-01T21:56:14.571 INFO:teuthology.orchestra.run.smithi170.stdout:(4/23): ceph-radosgw-16.2.14-328.g0dc074ce.el8. 7.2 MB/s | 12 MB 00:01 2023-12-01T21:56:14.589 INFO:teuthology.orchestra.run.smithi170.stdout:(5/23): libradosstriper1-16.2.14-328.g0dc074ce. 6.3 MB/s | 493 kB 00:00 2023-12-01T21:56:14.690 INFO:teuthology.orchestra.run.smithi170.stdout:(6/23): python3-ceph-argparse-16.2.14-328.g0dc0 448 kB/s | 45 kB 00:00 2023-12-01T21:56:14.785 INFO:teuthology.orchestra.run.smithi170.stdout:(7/23): python3-ceph-common-16.2.14-328.g0dc074 1.0 MB/s | 99 kB 00:00 2023-12-01T21:56:14.992 INFO:teuthology.orchestra.run.smithi170.stdout:(8/23): librgw2-16.2.14-328.g0dc074ce.el8.x86_6 9.0 MB/s | 3.8 MB 00:00 2023-12-01T21:56:15.086 INFO:teuthology.orchestra.run.smithi170.stdout:(9/23): python3-rados-16.2.14-328.g0dc074ce.el8 4.1 MB/s | 390 kB 00:00 2023-12-01T21:56:15.160 INFO:teuthology.orchestra.run.smithi170.stdout:(10/23): python3-rbd-16.2.14-328.g0dc074ce.el8. 4.9 MB/s | 369 kB 00:00 2023-12-01T21:56:15.238 INFO:teuthology.orchestra.run.smithi170.stdout:(11/23): python3-rgw-16.2.14-328.g0dc074ce.el8. 1.4 MB/s | 114 kB 00:00 2023-12-01T21:56:15.502 INFO:teuthology.orchestra.run.smithi170.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-12-01T21:56:15.541 INFO:teuthology.orchestra.run.smithi170.stdout:(13/23): leveldb-1.22-1.el8.x86_64.rpm 4.5 MB/s | 181 kB 00:00 2023-12-01T21:56:15.546 INFO:teuthology.orchestra.run.smithi039.stdout:(1/23): ceph-base-16.2.14-328.g0dc074ce.el8.x86 4.1 MB/s | 6.1 MB 00:01 2023-12-01T21:56:15.576 INFO:teuthology.orchestra.run.smithi170.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 1.6 MB/s | 59 kB 00:00 2023-12-01T21:56:15.604 INFO:teuthology.orchestra.run.smithi039.stdout:(2/23): ceph-selinux-16.2.14-328.g0dc074ce.el8. 448 kB/s | 26 kB 00:00 2023-12-01T21:56:15.611 INFO:teuthology.orchestra.run.smithi170.stdout:(15/23): python3-cephfs-16.2.14-328.g0dc074ce.e 261 kB/s | 215 kB 00:00 2023-12-01T21:56:15.615 INFO:teuthology.orchestra.run.smithi170.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.9 MB/s | 75 kB 00:00 2023-12-01T21:56:15.656 INFO:teuthology.orchestra.run.smithi170.stdout:(17/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.0 MB/s | 47 kB 00:00 2023-12-01T21:56:15.660 INFO:teuthology.orchestra.run.smithi170.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 990 kB/s | 44 kB 00:00 2023-12-01T21:56:15.686 INFO:teuthology.orchestra.run.smithi170.stdout:(19/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.4 MB/s | 259 kB 00:00 2023-12-01T21:56:15.690 INFO:teuthology.orchestra.run.smithi170.stdout:(20/23): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2023-12-01T21:56:15.747 INFO:teuthology.orchestra.run.smithi039.stdout:(3/23): libcephfs2-16.2.14-328.g0dc074ce.el8.x8 5.6 MB/s | 825 kB 00:00 2023-12-01T21:56:16.298 INFO:teuthology.orchestra.run.smithi170.stdout:(21/23): librados2-16.2.14-328.g0dc074ce.el8.x8 6.1 MB/s | 3.7 MB 00:00 2023-12-01T21:56:16.468 INFO:teuthology.orchestra.run.smithi039.stdout:(4/23): libradosstriper1-16.2.14-328.g0dc074ce. 685 kB/s | 493 kB 00:00 2023-12-01T21:56:16.502 INFO:teuthology.orchestra.run.smithi170.stdout:(22/23): librbd1-16.2.14-328.g0dc074ce.el8.x86_ 5.1 MB/s | 4.2 MB 00:00 2023-12-01T21:56:16.691 INFO:teuthology.orchestra.run.smithi170.stdout:(23/23): ceph-common-16.2.14-328.g0dc074ce.el8. 6.5 MB/s | 24 MB 00:03 2023-12-01T21:56:16.695 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:56:16.695 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 57 MB 00:04 2023-12-01T21:56:16.700 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:56:16.745 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:56:16.745 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:56:16.847 INFO:teuthology.orchestra.run.smithi039.stdout:(5/23): librgw2-16.2.14-328.g0dc074ce.el8.x86_6 10 MB/s | 3.8 MB 00:00 2023-12-01T21:56:16.889 INFO:teuthology.orchestra.run.smithi039.stdout:(6/23): python3-ceph-argparse-16.2.14-328.g0dc0 1.1 MB/s | 45 kB 00:00 2023-12-01T21:56:16.932 INFO:teuthology.orchestra.run.smithi039.stdout:(7/23): python3-ceph-common-16.2.14-328.g0dc074 2.3 MB/s | 99 kB 00:00 2023-12-01T21:56:16.976 INFO:teuthology.orchestra.run.smithi039.stdout:(8/23): python3-cephfs-16.2.14-328.g0dc074ce.el 4.8 MB/s | 215 kB 00:00 2023-12-01T21:56:17.020 INFO:teuthology.orchestra.run.smithi039.stdout:(9/23): python3-rados-16.2.14-328.g0dc074ce.el8 8.6 MB/s | 390 kB 00:00 2023-12-01T21:56:17.084 INFO:teuthology.orchestra.run.smithi039.stdout:(10/23): ceph-radosgw-16.2.14-328.g0dc074ce.el8 3.8 MB/s | 12 MB 00:03 2023-12-01T21:56:17.088 INFO:teuthology.orchestra.run.smithi039.stdout:(11/23): python3-rbd-16.2.14-328.g0dc074ce.el8. 5.4 MB/s | 369 kB 00:00 2023-12-01T21:56:17.114 INFO:teuthology.orchestra.run.smithi039.stdout:(12/23): python3-rgw-16.2.14-328.g0dc074ce.el8. 3.8 MB/s | 114 kB 00:00 2023-12-01T21:56:17.507 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:56:17.511 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:56:17.752 INFO:teuthology.orchestra.run.smithi039.stdout:(13/23): ceph-common-16.2.14-328.g0dc074ce.el8. 6.6 MB/s | 24 MB 00:03 2023-12-01T21:56:17.907 INFO:teuthology.orchestra.run.smithi039.stdout:(14/23): leveldb-1.22-1.el8.x86_64.rpm 228 kB/s | 181 kB 00:00 2023-12-01T21:56:17.916 INFO:teuthology.orchestra.run.smithi039.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 370 kB/s | 306 kB 00:00 2023-12-01T21:56:17.947 INFO:teuthology.orchestra.run.smithi039.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2023-12-01T21:56:17.961 INFO:teuthology.orchestra.run.smithi039.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 17 MB/s | 259 kB 00:00 2023-12-01T21:56:17.978 INFO:teuthology.orchestra.run.smithi039.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 22 MB/s | 353 kB 00:00 2023-12-01T21:56:17.989 INFO:teuthology.orchestra.run.smithi039.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.9 MB/s | 47 kB 00:00 2023-12-01T21:56:18.089 INFO:teuthology.orchestra.run.smithi039.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 412 kB/s | 75 kB 00:00 2023-12-01T21:56:18.121 INFO:teuthology.orchestra.run.smithi039.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 159 kB/s | 59 kB 00:00 2023-12-01T21:56:18.388 INFO:teuthology.orchestra.run.smithi039.stdout:(22/23): librados2-16.2.14-328.g0dc074ce.el8.x8 9.4 MB/s | 3.7 MB 00:00 2023-12-01T21:56:18.574 INFO:teuthology.orchestra.run.smithi039.stdout:(23/23): librbd1-16.2.14-328.g0dc074ce.el8.x86_ 8.6 MB/s | 4.2 MB 00:00 2023-12-01T21:56:18.577 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:56:18.577 INFO:teuthology.orchestra.run.smithi039.stdout:Total 13 MB/s | 57 MB 00:04 2023-12-01T21:56:18.582 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:56:18.590 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:56:18.633 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:56:18.634 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:56:18.798 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-01T21:56:19.187 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-01T21:56:19.349 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:56:19.353 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:56:19.440 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 2/25 2023-12-01T21:56:19.528 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 2/25 2023-12-01T21:56:19.746 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 3/25 2023-12-01T21:56:19.997 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-12-01T21:56:20.206 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 5/25 2023-12-01T21:56:20.488 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:56:20.622 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 5/25 2023-12-01T21:56:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-01T21:56:20.831 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 6/25 2023-12-01T21:56:20.915 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 6/25 2023-12-01T21:56:21.083 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-01T21:56:21.148 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-12-01T21:56:21.219 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-12-01T21:56:21.308 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 2/25 2023-12-01T21:56:21.414 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 2/25 2023-12-01T21:56:21.638 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 3/25 2023-12-01T21:56:21.721 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-12-01T21:56:21.894 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-12-01T21:56:21.939 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 9/25 2023-12-01T21:56:22.017 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 9/25 2023-12-01T21:56:22.083 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 5/25 2023-12-01T21:56:22.229 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-01T21:56:22.354 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:16.2.14-328.g0dc074ce.el8. 11/25 2023-12-01T21:56:22.513 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 5/25 2023-12-01T21:56:22.576 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 12/25 2023-12-01T21:56:22.724 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 6/25 2023-12-01T21:56:22.792 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 6/25 2023-12-01T21:56:22.824 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 13/25 2023-12-01T21:56:22.973 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-12-01T21:56:23.090 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 14/25 2023-12-01T21:56:23.174 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-12-01T21:56:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_6 15/25 2023-12-01T21:56:23.370 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_6 15/25 2023-12-01T21:56:23.558 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-12-01T21:56:23.560 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-01T21:56:23.732 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 9/25 2023-12-01T21:56:23.809 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-01T21:56:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 9/25 2023-12-01T21:56:24.006 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-01T21:56:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-01T21:56:24.148 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:16.2.14-328.g0dc074ce.el8. 11/25 2023-12-01T21:56:24.256 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:16.2.14-328.g0dc074ce.el8.x8 19/25 2023-12-01T21:56:24.388 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 12/25 2023-12-01T21:56:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 13/25 2023-12-01T21:56:24.828 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 14/25 2023-12-01T21:56:24.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_6 15/25 2023-12-01T21:56:25.045 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_6 15/25 2023-12-01T21:56:25.251 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-01T21:56:25.480 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-01T21:56:25.734 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-01T21:56:25.916 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:16.2.14-328.g0dc074ce.el8.x8 19/25 2023-12-01T21:56:29.299 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 20/25 2023-12-01T21:56:29.530 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 20/25 2023-12-01T21:56:30.252 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 20/25 2023-12-01T21:56:30.576 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 21/25 2023-12-01T21:56:30.684 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 21/25 2023-12-01T21:56:30.796 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 22/25 2023-12-01T21:56:31.253 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 20/25 2023-12-01T21:56:31.482 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 20/25 2023-12-01T21:56:32.253 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 20/25 2023-12-01T21:56:32.556 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 21/25 2023-12-01T21:56:32.664 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 21/25 2023-12-01T21:56:32.773 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 22/25 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 22/25 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /dev 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /run 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2023-12-01T21:56:48.260 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2023-12-01T21:56:48.261 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2023-12-01T21:56:48.261 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2023-12-01T21:56:48.261 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /dev 2023-12-01T21:56:48.261 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:49.498 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 23/25 2023-12-01T21:56:49.579 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 23/25 2023-12-01T21:56:49.579 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-01T21:56:49.812 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-01T21:56:49.812 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-01T21:56:50.643 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 1/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 2/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 3/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 4/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 5/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_6 6/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 7/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:16.2.14-328.g0dc074ce.el8. 8/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:16.2.14-328.g0dc074ce.el8.x8 9/25 2023-12-01T21:56:50.644 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 10/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 11/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 12/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 13/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-12-01T21:56:50.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-12-01T21:56:50.646 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-12-01T21:56:50.646 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-12-01T21:56:50.646 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 22/25 2023-12-01T21:56:50.646 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-01T21:56:50.646 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 24/25 2023-12-01T21:56:50.887 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 22/25 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /run 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-12-01T21:56:50.888 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:56:51.997 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-01T21:56:51.997 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:47 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:56:52.128 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 23/25 2023-12-01T21:56:52.203 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 23/25 2023-12-01T21:56:52.203 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-01T21:56:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-01T21:56:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-01T21:56:53.231 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-01T21:56:53.231 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 1/25 2023-12-01T21:56:53.231 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 2/25 2023-12-01T21:56:53.231 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 3/25 2023-12-01T21:56:53.231 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 4/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 5/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_6 6/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 7/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:16.2.14-328.g0dc074ce.el8. 8/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:16.2.14-328.g0dc074ce.el8.x8 9/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 10/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 11/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 12/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 13/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-01T21:56:53.232 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-01T21:56:53.233 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-12-01T21:56:53.233 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-12-01T21:56:53.233 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-12-01T21:56:53.233 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 22/25 2023-12-01T21:56:53.233 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-01T21:56:53.233 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 24/25 2023-12-01T21:56:53.892 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:56:54.001 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:54.001 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: leveldb-1.22-1.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-01T21:56:54.002 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:54.003 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:54.004 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:56:54.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2023-12-01T21:56:54.513 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:56:54.680 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-01T21:56:54.680 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:50 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:56:55.275 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:51 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:56:56.676 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:56:56.795 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:56:56.795 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: leveldb-1.22-1.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.796 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:56:56.797 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:56:56.938 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2023-12-01T21:56:57.057 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 52 M 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:56:57.058 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 174 k 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-01T21:56:57.059 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:56:57.060 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 53 M 2023-12-01T21:56:57.060 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 196 M 2023-12-01T21:56:57.060 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:56:57.064 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 17 MB/s | 70 kB 00:00 2023-12-01T21:56:57.119 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 2.9 MB/s | 161 kB 00:00 2023-12-01T21:56:57.152 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.6 MB/s | 187 kB 00:00 2023-12-01T21:56:57.184 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 9.4 MB/s | 302 kB 00:00 2023-12-01T21:56:57.229 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): libcephsqlite-16.2.14-328.g0dc074ce.el8. 1.0 MB/s | 174 kB 00:00 2023-12-01T21:56:57.326 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:56:58.270 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:54 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:56:59.457 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-16.2.14-328.g0dc074ce.el8.x86_ 22 MB/s | 52 MB 00:02 2023-12-01T21:56:59.458 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:56:59.459 INFO:teuthology.orchestra.run.smithi170.stdout:Total 22 MB/s | 53 MB 00:02 2023-12-01T21:56:59.462 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:56:59.486 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:56:59.486 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:56:59.880 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:56:59.884 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:57:00.230 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:57:00.231 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2023-12-01T21:57:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:57:00.231 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2023-12-01T21:57:00.231 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 52 M 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 174 k 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-12-01T21:57:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-12-01T21:57:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:00.233 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:57:00.233 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2023-12-01T21:57:00.233 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2023-12-01T21:57:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:00.234 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 53 M 2023-12-01T21:57:00.234 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 196 M 2023-12-01T21:57:00.234 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:57:00.237 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-12-01T21:57:00.280 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.7 MB/s | 161 kB 00:00 2023-12-01T21:57:00.298 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 10 MB/s | 187 kB 00:00 2023-12-01T21:57:00.319 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-12-01T21:57:00.392 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): libcephsqlite-16.2.14-328.g0dc074ce.el8. 1.1 MB/s | 174 kB 00:00 2023-12-01T21:57:00.418 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:57:00.670 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-12-01T21:57:00.883 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-01T21:57:00.978 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-01T21:57:01.151 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-12-01T21:57:01.356 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-12-01T21:57:01.516 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 5/6 2023-12-01T21:57:03.577 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): ceph-test-16.2.14-328.g0dc074ce.el8.x86_ 15 MB/s | 52 MB 00:03 2023-12-01T21:57:03.580 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:03.580 INFO:teuthology.orchestra.run.smithi039.stdout:Total 16 MB/s | 53 MB 00:03 2023-12-01T21:57:03.584 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:57:03.610 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:57:03.610 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:57:03.982 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:57:03.986 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:57:04.541 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:57:04.729 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-12-01T21:57:04.951 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-01T21:57:05.035 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-01T21:57:05.186 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-12-01T21:57:05.375 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-12-01T21:57:05.518 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 5/6 2023-12-01T21:57:06.141 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 5/6 2023-12-01T21:57:06.439 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 6/6 2023-12-01T21:57:06.701 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 6/6 2023-12-01T21:57:06.701 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 1/6 2023-12-01T21:57:06.701 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 2/6 2023-12-01T21:57:06.701 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-12-01T21:57:06.702 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-12-01T21:57:06.702 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-12-01T21:57:07.710 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-12-01T21:57:07.710 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:03 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:09.554 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:57:09.651 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:09.651 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:57:09.651 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:09.651 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.5-12.el8.x86_64 2023-12-01T21:57:09.652 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:09.652 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-01T21:57:09.652 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.3.3-2.el8.x86_64 2023-12-01T21:57:09.652 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-01T21:57:09.652 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:09.652 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:57:09.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2023-12-01T21:57:10.121 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:10.443 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 5/6 2023-12-01T21:57:10.701 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 6/6 2023-12-01T21:57:10.870 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:06 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:10.978 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 6/6 2023-12-01T21:57:10.978 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 1/6 2023-12-01T21:57:10.978 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 2/6 2023-12-01T21:57:10.979 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-12-01T21:57:10.979 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-12-01T21:57:10.979 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-12-01T21:57:12.214 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-12-01T21:57:12.214 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:08 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:12.656 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:57:12.659 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================= 2023-12-01T21:57:12.659 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:57:12.659 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================= 2023-12-01T21:57:12.659 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:57:12.659 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 6.4 k 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 2.4 M 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 1.6 M 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 245 k 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 5.0 M 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 18 M 2023-12-01T21:57:12.660 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-01T21:57:12.661 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-01T21:57:12.662 INFO:teuthology.orchestra.run.smithi170.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-01T21:57:12.663 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:12.664 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:57:12.664 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================= 2023-12-01T21:57:12.664 INFO:teuthology.orchestra.run.smithi170.stdout:Install 28 Packages 2023-12-01T21:57:12.664 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:12.665 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 30 M 2023-12-01T21:57:12.665 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 97 M 2023-12-01T21:57:12.666 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:57:13.175 INFO:teuthology.orchestra.run.smithi170.stdout:(1/28): ceph-16.2.14-328.g0dc074ce.el8.x86_64.r 38 kB/s | 6.4 kB 00:00 2023-12-01T21:57:13.441 INFO:teuthology.orchestra.run.smithi170.stdout:(2/28): ceph-mgr-16.2.14-328.g0dc074ce.el8.x86_ 3.7 MB/s | 1.6 MB 00:00 2023-12-01T21:57:13.487 INFO:teuthology.orchestra.run.smithi170.stdout:(3/28): ceph-mds-16.2.14-328.g0dc074ce.el8.x86_ 5.1 MB/s | 2.4 MB 00:00 2023-12-01T21:57:13.586 INFO:teuthology.orchestra.run.smithi170.stdout:(4/28): ceph-mgr-modules-core-16.2.14-328.g0dc0 2.4 MB/s | 245 kB 00:00 2023-12-01T21:57:13.955 INFO:teuthology.orchestra.run.smithi170.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 118 kB/s | 44 kB 00:00 2023-12-01T21:57:14.076 INFO:teuthology.orchestra.run.smithi170.stdout:(6/28): ceph-mon-16.2.14-328.g0dc074ce.el8.x86_ 5.6 MB/s | 5.0 MB 00:00 2023-12-01T21:57:14.138 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.5-12.el8.x86_64 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.3.3-2.el8.x86_64 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-01T21:57:14.245 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:14.246 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:57:14.364 INFO:teuthology.orchestra.run.smithi170.stdout:(7/28): ceph-osd-16.2.14-328.g0dc074ce.el8.x86_ 20 MB/s | 18 MB 00:00 2023-12-01T21:57:14.364 INFO:teuthology.orchestra.run.smithi170.stdout:(8/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 452 kB/s | 185 kB 00:00 2023-12-01T21:57:14.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2023-12-01T21:57:14.456 INFO:teuthology.orchestra.run.smithi170.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 122 kB/s | 11 kB 00:00 2023-12-01T21:57:14.547 INFO:teuthology.orchestra.run.smithi170.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 193 kB/s | 18 kB 00:00 2023-12-01T21:57:14.638 INFO:teuthology.orchestra.run.smithi170.stdout:(11/28): python3-logutils-0.3.5-11.el8.noarch.r 533 kB/s | 49 kB 00:00 2023-12-01T21:57:14.730 INFO:teuthology.orchestra.run.smithi170.stdout:(12/28): python3-more-itertools-7.2.0-3.el8.noa 642 kB/s | 59 kB 00:00 2023-12-01T21:57:14.743 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:14.801 INFO:teuthology.orchestra.run.smithi170.stdout:(13/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 238 kB/s | 173 kB 00:00 2023-12-01T21:57:14.891 INFO:teuthology.orchestra.run.smithi170.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 172 kB/s | 16 kB 00:00 2023-12-01T21:57:14.982 INFO:teuthology.orchestra.run.smithi170.stdout:(15/28): python3-simplegeneric-0.8.1-17.el8.noa 206 kB/s | 19 kB 00:00 2023-12-01T21:57:15.074 INFO:teuthology.orchestra.run.smithi170.stdout:(16/28): python3-singledispatch-3.4.0.3-18.el8. 263 kB/s | 24 kB 00:00 2023-12-01T21:57:15.165 INFO:teuthology.orchestra.run.smithi170.stdout:(17/28): python3-tempora-1.14.1-5.el8.noarch.rp 321 kB/s | 29 kB 00:00 2023-12-01T21:57:15.184 INFO:teuthology.orchestra.run.smithi170.stdout:(18/28): python3-pecan-1.3.2-9.el8.noarch.rpm 623 kB/s | 283 kB 00:00 2023-12-01T21:57:15.256 INFO:teuthology.orchestra.run.smithi170.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 303 kB/s | 27 kB 00:00 2023-12-01T21:57:15.264 INFO:teuthology.orchestra.run.smithi170.stdout:(20/28): python3-cherrypy-18.4.0-1.el8.noarch.r 427 kB/s | 384 kB 00:00 2023-12-01T21:57:15.446 INFO:teuthology.orchestra.run.smithi170.stdout:(21/28): python3-webtest-2.0.33-1.el8.noarch.rp 470 kB/s | 85 kB 00:00 2023-12-01T21:57:15.536 INFO:teuthology.orchestra.run.smithi170.stdout:(22/28): python3-zc-lockfile-2.0-2.el8.noarch.r 254 kB/s | 23 kB 00:00 2023-12-01T21:57:15.547 INFO:teuthology.orchestra.run.smithi170.stdout:(23/28): python3-waitress-1.4.3-1.el8.noarch.rp 703 kB/s | 255 kB 00:00 2023-12-01T21:57:15.604 INFO:teuthology.orchestra.run.smithi170.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 6.6 MB/s | 457 kB 00:00 2023-12-01T21:57:15.604 INFO:teuthology.orchestra.run.smithi170.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 676 kB/s | 39 kB 00:00 2023-12-01T21:57:15.621 INFO:teuthology.orchestra.run.smithi170.stdout:(26/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 689 kB/s | 251 kB 00:00 2023-12-01T21:57:15.642 INFO:teuthology.orchestra.run.smithi170.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 4.0 MB/s | 157 kB 00:00 2023-12-01T21:57:15.647 INFO:teuthology.orchestra.run.smithi170.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 1.2 MB/s | 54 kB 00:00 2023-12-01T21:57:15.650 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:15.651 INFO:teuthology.orchestra.run.smithi170.stdout:Total 10 MB/s | 30 MB 00:02 2023-12-01T21:57:15.655 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:57:15.693 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:57:15.693 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:57:15.735 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:11 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:16.079 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:57:16.084 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:57:17.148 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:57:17.404 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-12-01T21:57:17.656 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================================= 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================================= 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 6.4 k 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:57:17.659 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 2.4 M 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 1.6 M 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 245 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 5.0 M 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 18 M 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-12-01T21:57:17.660 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-01T21:57:17.661 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================================= 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout:Install 28 Packages 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 30 M 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 97 M 2023-12-01T21:57:17.662 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:57:17.687 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-12-01T21:57:17.784 INFO:teuthology.orchestra.run.smithi039.stdout:(1/28): ceph-16.2.14-328.g0dc074ce.el8.x86_64.r 72 kB/s | 6.4 kB 00:00 2023-12-01T21:57:17.894 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-12-01T21:57:18.027 INFO:teuthology.orchestra.run.smithi039.stdout:(2/28): ceph-mds-16.2.14-328.g0dc074ce.el8.x86_ 7.3 MB/s | 2.4 MB 00:00 2023-12-01T21:57:18.111 INFO:teuthology.orchestra.run.smithi039.stdout:(3/28): ceph-mgr-16.2.14-328.g0dc074ce.el8.x86_ 3.9 MB/s | 1.6 MB 00:00 2023-12-01T21:57:18.181 INFO:teuthology.orchestra.run.smithi039.stdout:(4/28): ceph-mgr-modules-core-16.2.14-328.g0dc0 3.4 MB/s | 245 kB 00:00 2023-12-01T21:57:18.190 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-12-01T21:57:18.322 INFO:teuthology.orchestra.run.smithi039.stdout:(5/28): ceph-mon-16.2.14-328.g0dc074ce.el8.x86_ 9.4 MB/s | 5.0 MB 00:00 2023-12-01T21:57:18.322 INFO:teuthology.orchestra.run.smithi039.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 309 kB/s | 44 kB 00:00 2023-12-01T21:57:18.419 INFO:teuthology.orchestra.run.smithi039.stdout:(7/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.9 MB/s | 185 kB 00:00 2023-12-01T21:57:18.492 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-12-01T21:57:18.494 INFO:teuthology.orchestra.run.smithi039.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 5.0 MB/s | 384 kB 00:00 2023-12-01T21:57:18.524 INFO:teuthology.orchestra.run.smithi039.stdout:(9/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 854 kB/s | 173 kB 00:00 2023-12-01T21:57:18.537 INFO:teuthology.orchestra.run.smithi039.stdout:(10/28): python3-jaraco-6.2-6.el8.noarch.rpm 263 kB/s | 11 kB 00:00 2023-12-01T21:57:18.567 INFO:teuthology.orchestra.run.smithi039.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 416 kB/s | 18 kB 00:00 2023-12-01T21:57:18.575 INFO:teuthology.orchestra.run.smithi039.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 1.3 MB/s | 49 kB 00:00 2023-12-01T21:57:18.621 INFO:teuthology.orchestra.run.smithi039.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-12-01T21:57:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 4.4 MB/s | 283 kB 00:00 2023-12-01T21:57:18.677 INFO:teuthology.orchestra.run.smithi039.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 276 kB/s | 16 kB 00:00 2023-12-01T21:57:18.706 INFO:teuthology.orchestra.run.smithi039.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 279 kB/s | 19 kB 00:00 2023-12-01T21:57:18.738 INFO:teuthology.orchestra.run.smithi039.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 397 kB/s | 24 kB 00:00 2023-12-01T21:57:18.747 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-12-01T21:57:18.771 INFO:teuthology.orchestra.run.smithi039.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 450 kB/s | 29 kB 00:00 2023-12-01T21:57:18.912 INFO:teuthology.orchestra.run.smithi039.stdout:(19/28): ceph-osd-16.2.14-328.g0dc074ce.el8.x86 21 MB/s | 18 MB 00:00 2023-12-01T21:57:18.913 INFO:teuthology.orchestra.run.smithi039.stdout:(20/28): python3-trustme-0.6.0-4.el8.noarch.rpm 157 kB/s | 27 kB 00:00 2023-12-01T21:57:18.988 INFO:teuthology.orchestra.run.smithi039.stdout:(21/28): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2023-12-01T21:57:18.989 INFO:teuthology.orchestra.run.smithi039.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-12-01T21:57:19.011 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-12-01T21:57:19.031 INFO:teuthology.orchestra.run.smithi039.stdout:(23/28): python3-werkzeug-0.12.2-4.el8.noarch.r 11 MB/s | 457 kB 00:00 2023-12-01T21:57:19.045 INFO:teuthology.orchestra.run.smithi039.stdout:(24/28): python3-markupsafe-0.23-19.el8.x86_64. 2.8 MB/s | 39 kB 00:00 2023-12-01T21:57:19.060 INFO:teuthology.orchestra.run.smithi039.stdout:(25/28): python3-mako-1.0.6-13.el8.noarch.rpm 10 MB/s | 157 kB 00:00 2023-12-01T21:57:19.064 INFO:teuthology.orchestra.run.smithi039.stdout:(26/28): python3-zc-lockfile-2.0-2.el8.noarch.r 301 kB/s | 23 kB 00:00 2023-12-01T21:57:19.073 INFO:teuthology.orchestra.run.smithi039.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 4.0 MB/s | 54 kB 00:00 2023-12-01T21:57:19.205 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-12-01T21:57:19.319 INFO:teuthology.orchestra.run.smithi039.stdout:(28/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 618 kB/s | 251 kB 00:00 2023-12-01T21:57:19.322 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:19.322 INFO:teuthology.orchestra.run.smithi039.stdout:Total 18 MB/s | 30 MB 00:01 2023-12-01T21:57:19.327 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:57:19.371 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:57:19.371 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:57:19.412 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-12-01T21:57:19.627 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-12-01T21:57:19.719 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:57:19.723 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:57:19.809 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-12-01T21:57:20.033 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-12-01T21:57:20.195 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-12-01T21:57:20.435 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-12-01T21:57:20.651 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-12-01T21:57:20.786 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:57:20.853 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-12-01T21:57:21.004 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-12-01T21:57:21.102 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-12-01T21:57:21.248 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-12-01T21:57:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-12-01T21:57:21.421 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-12-01T21:57:21.599 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-12-01T21:57:21.693 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-12-01T21:57:21.958 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-12-01T21:57:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-12-01T21:57:22.208 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-12-01T21:57:22.239 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-12-01T21:57:22.456 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-12-01T21:57:22.565 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-12-01T21:57:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-12-01T21:57:22.817 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-12-01T21:57:22.880 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-328.g0dc074ce.el8. 23/28 2023-12-01T21:57:22.979 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-12-01T21:57:23.145 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-12-01T21:57:23.352 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-12-01T21:57:23.385 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 24/28 2023-12-01T21:57:23.489 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-12-01T21:57:23.747 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-12-01T21:57:23.946 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-12-01T21:57:24.147 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-12-01T21:57:24.404 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-12-01T21:57:24.667 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-12-01T21:57:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 24/28 2023-12-01T21:57:24.923 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-12-01T21:57:25.104 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 25/28 2023-12-01T21:57:25.234 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-12-01T21:57:25.531 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 25/28 2023-12-01T21:57:25.533 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-12-01T21:57:25.721 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 26/28 2023-12-01T21:57:25.825 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-12-01T21:57:25.940 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 26/28 2023-12-01T21:57:26.121 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 27/28 2023-12-01T21:57:26.140 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 27/28 2023-12-01T21:57:26.254 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 28/28 2023-12-01T21:57:26.461 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-328.g0dc074ce.el8. 23/28 2023-12-01T21:57:26.646 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 28/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 1/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 2/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 3/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 4/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 5/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-328.g0dc074ce.el8. 6/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-12-01T21:57:26.647 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-12-01T21:57:26.650 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-12-01T21:57:26.651 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-12-01T21:57:26.652 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-12-01T21:57:26.652 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-12-01T21:57:26.652 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-12-01T21:57:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 24/28 2023-12-01T21:57:28.136 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-12-01T21:57:28.136 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:24 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:28.262 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 24/28 2023-12-01T21:57:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 25/28 2023-12-01T21:57:28.886 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 25/28 2023-12-01T21:57:29.057 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 26/28 2023-12-01T21:57:29.289 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 26/28 2023-12-01T21:57:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 27/28 2023-12-01T21:57:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 27/28 2023-12-01T21:57:29.548 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 28/28 2023-12-01T21:57:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 28/28 2023-12-01T21:57:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 1/28 2023-12-01T21:57:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 2/28 2023-12-01T21:57:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 3/28 2023-12-01T21:57:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 4/28 2023-12-01T21:57:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 5/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-328.g0dc074ce.el8. 6/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-12-01T21:57:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-12-01T21:57:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-12-01T21:57:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-12-01T21:57:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-12-01T21:57:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-12-01T21:57:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-12-01T21:57:29.989 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-01T21:57:30.089 INFO:teuthology.orchestra.run.smithi170.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.0.6-13.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-2.6-1.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2017.2-9.el8.noarch 2023-12-01T21:57:30.090 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:30.091 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:57:30.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2023-12-01T21:57:30.589 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:31.375 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:27 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:31.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-12-01T21:57:31.531 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:27 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:32.682 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:57:33.168 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:57:33.169 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:57:33.169 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:57:33.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2023-12-01T21:57:33.627 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:33.661 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-01T21:57:33.770 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.0.6-13.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-2.6-1.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2017.2-9.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-01T21:57:33.771 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-01T21:57:33.772 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-01T21:57:33.772 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-01T21:57:33.772 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-01T21:57:33.772 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-01T21:57:33.772 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:33.772 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:57:33.939 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2023-12-01T21:57:34.327 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:35.226 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:31 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:36.085 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:32 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:37.021 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:57:37.021 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 85 k 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 85 k 2023-12-01T21:57:37.022 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 352 k 2023-12-01T21:57:37.023 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:57:37.362 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-16.2.14-328.g0dc074ce.el8.noarch.rpm 251 kB/s | 85 kB 00:00 2023-12-01T21:57:37.363 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:37.363 INFO:teuthology.orchestra.run.smithi170.stdout:Total 250 kB/s | 85 kB 00:00 2023-12-01T21:57:37.363 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:57:37.370 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:57:37.370 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:57:37.372 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:57:37.377 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:57:37.456 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:57:37.483 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:57:37.990 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:57:37.991 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:57:37.992 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:57:38.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2023-12-01T21:57:38.481 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:40.165 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:36 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:41.193 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:41.353 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:41.557 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:42.045 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 85 k 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:42.046 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:57:42.047 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:57:42.047 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T21:57:42.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:42.047 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 85 k 2023-12-01T21:57:42.047 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 352 k 2023-12-01T21:57:42.047 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:57:42.339 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-16.2.14-328.g0dc074ce.el8.noarch.rpm 292 kB/s | 85 kB 00:00 2023-12-01T21:57:42.340 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:42.341 INFO:teuthology.orchestra.run.smithi039.stdout:Total 290 kB/s | 85 kB 00:00 2023-12-01T21:57:42.341 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:57:42.349 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:57:42.349 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:57:42.351 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:57:42.356 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:57:42.510 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:42.510 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:38 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:43.105 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:57:44.655 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:57:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:44.751 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:57:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:57:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:44.751 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:57:44.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2023-12-01T21:57:45.252 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:46.120 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:42 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:46.469 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:46.634 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:46.844 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:47.935 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 161 k 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:57:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:57:47.937 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:47.937 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 161 k 2023-12-01T21:57:47.937 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 470 k 2023-12-01T21:57:47.937 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:57:47.982 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/1 2023-12-01T21:57:47.982 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:43 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:48.089 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-16.2.14-328.g0dc074 1.0 MB/s | 161 kB 00:00 2023-12-01T21:57:48.090 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:48.091 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.0 MB/s | 161 kB 00:00 2023-12-01T21:57:48.091 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:57:48.103 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:57:48.104 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:57:48.156 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:57:48.160 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:57:48.284 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:57:48.459 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce. 1/1 2023-12-01T21:57:48.871 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce. 1/1 2023-12-01T21:57:49.859 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce. 1/1 2023-12-01T21:57:49.859 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:50.885 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:57:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:50.991 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:57:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:57:50.991 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:50.991 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:57:51.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2023-12-01T21:57:51.535 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:53.311 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:49 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:53.419 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:57:53.514 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:53.514 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:57:53.515 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:53.515 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:57:53.515 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:57:53.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2023-12-01T21:57:53.995 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:55.221 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:57:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:57:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 161 k 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:57:55.225 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T21:57:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:55.226 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 161 k 2023-12-01T21:57:55.226 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 470 k 2023-12-01T21:57:55.226 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:57:55.376 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-16.2.14-328.g0dc074 1.0 MB/s | 161 kB 00:00 2023-12-01T21:57:55.377 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:57:55.377 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.0 MB/s | 161 kB 00:00 2023-12-01T21:57:55.377 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:57:55.391 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:57:55.392 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:57:55.448 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:57:55.453 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:57:55.601 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:57:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce. 1/1 2023-12-01T21:57:55.879 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:51 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce. 1/1 2023-12-01T21:57:57.254 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce. 1/1 2023-12-01T21:57:57.255 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:53 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:57.391 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:57:57.862 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:57:57.862 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:57:57.863 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:57:57.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2023-12-01T21:57:58.326 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:57:59.605 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:57:59.627 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:55 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:57:59.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:59.713 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:57:59.713 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:57:59.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:57:59.713 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:58:00.554 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2023-12-01T21:58:00.945 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:01.415 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:=================================================================================================== 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:=================================================================================================== 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 3.1 M 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 24 k 2023-12-01T21:58:01.416 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 15 k 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout:=================================================================================================== 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-01T21:58:01.417 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:01.418 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.4 M 2023-12-01T21:58:01.418 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 22 M 2023-12-01T21:58:01.418 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:58:01.667 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): ceph-grafana-dashboards-16.2.14-328.g0dc 212 kB/s | 24 kB 00:00 2023-12-01T21:58:01.745 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-prometheus-alerts-16.2.14-328.g0dc0 77 kB/s | 15 kB 00:00 2023-12-01T21:58:01.852 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): ceph-mgr-dashboard-16.2.14-328.g0dc074ce 10 MB/s | 3.1 MB 00:00 2023-12-01T21:58:01.908 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 761 kB/s | 43 kB 00:00 2023-12-01T21:58:02.035 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 91 kB/s | 34 kB 00:00 2023-12-01T21:58:02.162 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:58 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:02.435 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 285 kB/s | 196 kB 00:00 2023-12-01T21:58:02.437 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:02.438 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.4 MB/s | 3.4 MB 00:01 2023-12-01T21:58:02.440 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:58:02.448 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:58:02.449 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:58:02.624 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:58:02.628 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:58:02.961 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:58:03.150 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-12-01T21:58:03.422 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-12-01T21:58:03.584 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-12-01T21:58:03.645 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:58:03.754 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-328.g0dc074ce.el8.n 4/6 2023-12-01T21:58:04.145 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:58:04.146 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:58:04.147 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:58:04.243 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2023-12-01T21:58:04.627 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:04.840 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-328.g0dc074ce.el8. 5/6 2023-12-01T21:58:05.136 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarc 6/6 2023-12-01T21:58:05.403 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarc 6/6 2023-12-01T21:58:05.403 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-328.g0dc074ce.el8. 1/6 2023-12-01T21:58:05.403 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarc 2/6 2023-12-01T21:58:05.404 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-328.g0dc074ce.el8.n 3/6 2023-12-01T21:58:05.404 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-12-01T21:58:05.404 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-12-01T21:58:05.572 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:01 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:06.336 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-12-01T21:58:06.337 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:02 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:07.472 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:58:07.473 INFO:teuthology.orchestra.run.smithi039.stdout:=================================================================================================== 2023-12-01T21:58:07.473 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:58:07.473 INFO:teuthology.orchestra.run.smithi039.stdout:=================================================================================================== 2023-12-01T21:58:07.473 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:58:07.473 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 3.1 M 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 24 k 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 15 k 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout:=================================================================================================== 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2023-12-01T21:58:07.474 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:07.475 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.4 M 2023-12-01T21:58:07.475 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 22 M 2023-12-01T21:58:07.475 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:58:07.886 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): ceph-prometheus-alerts-16.2.14-328.g0dc0 73 kB/s | 15 kB 00:00 2023-12-01T21:58:07.971 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): ceph-grafana-dashboards-16.2.14-328.g0dc 84 kB/s | 24 kB 00:00 2023-12-01T21:58:07.982 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 350 kB/s | 34 kB 00:00 2023-12-01T21:58:08.013 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2023-12-01T21:58:08.137 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-12-01T21:58:08.157 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:58:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:08.254 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:58:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:08.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-01T21:58:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-01T21:58:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-01T21:58:08.255 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:08.255 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:58:08.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-01T21:58:08.742 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:09.428 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): ceph-mgr-dashboard-16.2.14-328.g0dc074ce 1.8 MB/s | 3.1 MB 00:01 2023-12-01T21:58:09.433 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:09.434 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.8 MB/s | 3.4 MB 00:01 2023-12-01T21:58:09.435 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:58:09.446 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:58:09.446 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:58:09.509 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:05 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:09.622 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:58:09.627 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:58:09.944 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:58:10.170 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-12-01T21:58:10.377 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-12-01T21:58:10.578 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-12-01T21:58:10.690 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-328.g0dc074ce.el8.n 4/6 2023-12-01T21:58:11.299 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:58:11.300 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================================== 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================================== 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 8.6 M 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-12-01T21:58:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:11.302 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:58:11.302 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================================================== 2023-12-01T21:58:11.302 INFO:teuthology.orchestra.run.smithi170.stdout:Install 8 Packages 2023-12-01T21:58:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:11.302 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2023-12-01T21:58:11.302 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 199 M 2023-12-01T21:58:11.303 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:58:11.352 INFO:teuthology.orchestra.run.smithi170.stdout:(1/8): libgfortran-8.4.1-1.el8.x86_64.rpm 13 MB/s | 639 kB 00:00 2023-12-01T21:58:11.355 INFO:teuthology.orchestra.run.smithi170.stdout:(2/8): libquadmath-8.4.1-1.el8.x86_64.rpm 3.2 MB/s | 169 kB 00:00 2023-12-01T21:58:11.388 INFO:teuthology.orchestra.run.smithi170.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 6.7 MB/s | 225 kB 00:00 2023-12-01T21:58:11.468 INFO:teuthology.orchestra.run.smithi170.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 47 MB/s | 3.7 MB 00:00 2023-12-01T21:58:11.635 INFO:teuthology.orchestra.run.smithi170.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 51 MB/s | 15 MB 00:00 2023-12-01T21:58:11.661 INFO:teuthology.orchestra.run.smithi170.stdout:(6/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-12-01T21:58:11.707 INFO:teuthology.orchestra.run.smithi170.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 64 MB/s | 4.6 MB 00:00 2023-12-01T21:58:11.862 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-328.g0dc074ce.el8. 5/6 2023-12-01T21:58:12.149 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarc 6/6 2023-12-01T21:58:12.407 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarc 6/6 2023-12-01T21:58:12.407 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-328.g0dc074ce.el8. 1/6 2023-12-01T21:58:12.407 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarc 2/6 2023-12-01T21:58:12.407 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-328.g0dc074ce.el8.n 3/6 2023-12-01T21:58:12.407 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-12-01T21:58:12.407 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-12-01T21:58:12.775 INFO:teuthology.orchestra.run.smithi170.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-32 5.8 MB/s | 8.6 MB 00:01 2023-12-01T21:58:12.781 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:12.782 INFO:teuthology.orchestra.run.smithi170.stdout:Total 25 MB/s | 37 MB 00:01 2023-12-01T21:58:12.786 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:58:12.809 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:58:12.809 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:58:13.190 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:58:13.196 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:58:13.482 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-12-01T21:58:13.482 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:09 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:13.822 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:58:13.980 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-12-01T21:58:14.120 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-12-01T21:58:14.296 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-12-01T21:58:14.849 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-12-01T21:58:15.038 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-12-01T21:58:15.437 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:15.545 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:58:15.556 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-12-01T21:58:15.662 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-01T21:58:15.762 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-12-01T21:58:16.050 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:16.489 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-12-01T21:58:16.863 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-12-01T21:58:16.942 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:12 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:17.129 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-12-01T21:58:18.857 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:58:18.858 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-12-01T21:58:18.858 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:58:18.858 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-12-01T21:58:18.858 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:58:18.858 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 8.6 M 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout:Install 8 Packages 2023-12-01T21:58:18.859 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:18.860 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 37 M 2023-12-01T21:58:18.860 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 199 M 2023-12-01T21:58:18.860 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:58:18.912 INFO:teuthology.orchestra.run.smithi039.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 4.4 MB/s | 225 kB 00:00 2023-12-01T21:58:18.988 INFO:teuthology.orchestra.run.smithi039.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 49 MB/s | 3.7 MB 00:00 2023-12-01T21:58:19.084 INFO:teuthology.orchestra.run.smithi039.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 50 MB/s | 4.7 MB 00:00 2023-12-01T21:58:19.177 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-12-01T21:58:19.215 INFO:teuthology.orchestra.run.smithi039.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 41 MB/s | 15 MB 00:00 2023-12-01T21:58:19.233 INFO:teuthology.orchestra.run.smithi039.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 9.5 MB/s | 169 kB 00:00 2023-12-01T21:58:19.257 INFO:teuthology.orchestra.run.smithi039.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-12-01T21:58:19.283 INFO:teuthology.orchestra.run.smithi039.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 23 MB/s | 4.6 MB 00:00 2023-12-01T21:58:19.502 INFO:teuthology.orchestra.run.smithi039.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-32 13 MB/s | 8.6 MB 00:00 2023-12-01T21:58:19.504 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:19.504 INFO:teuthology.orchestra.run.smithi039.stdout:Total 58 MB/s | 37 MB 00:00 2023-12-01T21:58:19.517 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:58:19.538 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:58:19.538 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:58:19.907 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:58:19.911 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:58:20.518 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:58:20.568 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-12-01T21:58:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-12-01T21:58:20.753 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074c 8/8 2023-12-01T21:58:20.813 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-12-01T21:58:20.948 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-12-01T21:58:21.051 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074c 8/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074c 1/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-12-01T21:58:21.052 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-12-01T21:58:21.557 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-12-01T21:58:21.698 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-12-01T21:58:22.023 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-12-01T21:58:22.023 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:18 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:22.252 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-12-01T21:58:22.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-12-01T21:58:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-12-01T21:58:23.564 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-12-01T21:58:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-12-01T21:58:23.855 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.12-1.el8.x86_64 2023-12-01T21:58:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-12-01T21:58:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-12-01T21:58:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-12-01T21:58:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-12-01T21:58:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:23.957 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:58:24.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2023-12-01T21:58:24.456 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:25.251 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:21 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:25.976 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-12-01T21:58:27.061 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:58:27.062 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================== 2023-12-01T21:58:27.062 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:58:27.062 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================== 2023-12-01T21:58:27.062 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:58:27.062 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 27 k 2023-12-01T21:58:27.062 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:58:27.063 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================================== 2023-12-01T21:58:27.064 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2023-12-01T21:58:27.064 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:27.064 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 1.9 M 2023-12-01T21:58:27.064 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-12-01T21:58:27.064 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:58:27.421 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-12-01T21:58:27.539 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): ceph-mgr-rook-16.2.14-328.g0dc074ce.el8 242 kB/s | 27 kB 00:00 2023-12-01T21:58:27.623 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074c 8/8 2023-12-01T21:58:27.934 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074c 8/8 2023-12-01T21:58:27.934 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074c 1/8 2023-12-01T21:58:27.934 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-12-01T21:58:27.934 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-12-01T21:58:27.934 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-12-01T21:58:27.935 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-12-01T21:58:27.935 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-12-01T21:58:27.935 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-12-01T21:58:28.550 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 14 kB/s | 16 kB 00:01 2023-12-01T21:58:28.558 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 29 kB/s | 33 kB 00:01 2023-12-01T21:58:28.621 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 75 kB/s | 81 kB 00:01 2023-12-01T21:58:28.802 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-rsa-4.9-2.el8.noarch.rpm 263 kB/s | 64 kB 00:00 2023-12-01T21:58:28.842 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.8 MB/s | 155 kB 00:00 2023-12-01T21:58:28.865 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.9 MB/s | 43 kB 00:00 2023-12-01T21:58:28.887 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2023-12-01T21:58:28.910 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.4 MB/s | 126 kB 00:00 2023-12-01T21:58:28.934 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 4.6 MB/s | 110 kB 00:00 2023-12-01T21:58:28.956 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 926 kB/s | 20 kB 00:00 2023-12-01T21:58:29.040 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-12-01T21:58:29.040 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:25 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:29.057 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 140 kB/s | 61 kB 00:00 2023-12-01T21:58:29.110 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2023-12-01T21:58:29.114 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:29.114 INFO:teuthology.orchestra.run.smithi170.stdout:Total 955 kB/s | 1.9 MB 00:02 2023-12-01T21:58:29.120 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:58:29.135 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:58:29.135 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:58:29.228 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:58:29.232 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:58:29.751 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:58:29.997 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-01T21:58:30.274 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-12-01T21:58:30.534 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-12-01T21:58:30.735 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-12-01T21:58:30.970 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:58:31.024 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.12-1.el8.x86_64 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-12-01T21:58:31.080 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-12-01T21:58:31.081 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-12-01T21:58:31.081 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:31.081 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:58:31.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2023-12-01T21:58:31.317 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-01T21:58:31.599 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-01T21:58:31.641 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:31.860 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-12-01T21:58:32.103 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-01T21:58:32.376 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-12-01T21:58:32.518 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:28 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:33.604 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-12-01T21:58:33.876 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-12-01T21:58:34.059 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 13/13 2023-12-01T21:58:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 13/13 2023-12-01T21:58:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 1/13 2023-12-01T21:58:34.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-12-01T21:58:34.314 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-12-01T21:58:34.462 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================================== 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================================== 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 27 k 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-12-01T21:58:34.464 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================================== 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2023-12-01T21:58:34.465 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:34.466 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 1.9 M 2023-12-01T21:58:34.466 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 24 M 2023-12-01T21:58:34.466 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:58:34.731 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 132 kB/s | 16 kB 00:00 2023-12-01T21:58:34.735 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 259 kB/s | 33 kB 00:00 2023-12-01T21:58:34.790 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.3 MB/s | 81 kB 00:00 2023-12-01T21:58:34.795 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): ceph-mgr-rook-16.2.14-328.g0dc074ce.el8 146 kB/s | 27 kB 00:00 2023-12-01T21:58:34.869 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): python3-kubernetes-11.0.0-6.el8.noarch. 8.7 MB/s | 1.2 MB 00:00 2023-12-01T21:58:34.887 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-rsa-4.9-2.el8.noarch.rpm 661 kB/s | 64 kB 00:00 2023-12-01T21:58:34.905 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 735 kB/s | 27 kB 00:00 2023-12-01T21:58:34.919 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 494 kB/s | 61 kB 00:00 2023-12-01T21:58:34.922 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 6.3 MB/s | 110 kB 00:00 2023-12-01T21:58:34.923 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.5 MB/s | 126 kB 00:00 2023-12-01T21:58:34.942 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-requests-oauthlib-1.0.0-1.el8. 2.2 MB/s | 43 kB 00:00 2023-12-01T21:58:34.943 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 7.2 MB/s | 155 kB 00:00 2023-12-01T21:58:34.959 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 505 kB/s | 20 kB 00:00 2023-12-01T21:58:34.962 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:34.962 INFO:teuthology.orchestra.run.smithi039.stdout:Total 3.8 MB/s | 1.9 MB 00:00 2023-12-01T21:58:34.968 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:58:34.983 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:58:34.984 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:58:35.071 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:58:35.076 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:58:35.348 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-12-01T21:58:35.348 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:31 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:35.583 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:58:35.818 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-01T21:58:36.081 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-12-01T21:58:36.345 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-12-01T21:58:36.663 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-12-01T21:58:36.946 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-12-01T21:58:37.112 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-12-01T21:58:37.288 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:58:37.328 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-01T21:58:37.388 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-01T21:58:37.389 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-01T21:58:37.389 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-01T21:58:37.389 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-01T21:58:37.389 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-01T21:58:37.389 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:37.389 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:58:37.546 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-12-01T21:58:37.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2023-12-01T21:58:37.798 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-01T21:58:37.922 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:38.017 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-12-01T21:58:38.714 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:34 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:39.288 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-12-01T21:58:39.547 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-12-01T21:58:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 13/13 2023-12-01T21:58:40.022 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 13/13 2023-12-01T21:58:40.022 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 1/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-01T21:58:40.023 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-12-01T21:58:40.492 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:58:40.493 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================================== 2023-12-01T21:58:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:58:40.493 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================================== 2023-12-01T21:58:40.493 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:58:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 116 k 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================================== 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-01T21:58:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:40.495 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 5.8 M 2023-12-01T21:58:40.495 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-12-01T21:58:40.495 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:58:40.715 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 148 kB/s | 21 kB 00:00 2023-12-01T21:58:40.724 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-mgr-cephadm-16.2.14-328.g0dc074ce.e 763 kB/s | 116 kB 00:00 2023-12-01T21:58:40.808 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-remoto-1.2.1-1.el8.noarch.rpm 529 kB/s | 49 kB 00:00 2023-12-01T21:58:40.832 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-babel-2.5.1-5.el8.noarch.rpm 44 MB/s | 4.8 MB 00:00 2023-12-01T21:58:40.838 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.3 MB/s | 355 kB 00:00 2023-12-01T21:58:40.875 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 7.9 MB/s | 538 kB 00:00 2023-12-01T21:58:40.877 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:40.877 INFO:teuthology.orchestra.run.smithi170.stdout:Total 15 MB/s | 5.8 MB 00:00 2023-12-01T21:58:40.878 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:58:40.887 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:58:40.887 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:58:40.998 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:58:41.002 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:58:41.230 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-12-01T21:58:41.230 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:37 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:42.030 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:58:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2023-12-01T21:58:42.648 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2023-12-01T21:58:42.974 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2023-12-01T21:58:43.206 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:58:43.306 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-01T21:58:43.314 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:43.315 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:58:43.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2023-12-01T21:58:43.593 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 5/6 2023-12-01T21:58:43.824 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 6/6 2023-12-01T21:58:43.875 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 6/6 2023-12-01T21:58:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/6 2023-12-01T21:58:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2023-12-01T21:58:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2023-12-01T21:58:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 4/6 2023-12-01T21:58:44.097 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 5/6 2023-12-01T21:58:44.800 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:40 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:45.045 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 6/6 2023-12-01T21:58:45.045 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:41 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:46.703 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================================== 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================================== 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:16.2.14-328.g0dc074ce.el8 ceph-noarch 116 k 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:58:46.704 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================================== 2023-12-01T21:58:46.705 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2023-12-01T21:58:46.705 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:46.705 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 5.8 M 2023-12-01T21:58:46.705 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 24 M 2023-12-01T21:58:46.705 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:58:47.055 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:58:47.145 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 227 kB/s | 21 kB 00:00 2023-12-01T21:58:47.153 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.5.1-5.el8.noarch 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:47.154 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:58:47.243 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.8 MB/s | 355 kB 00:00 2023-12-01T21:58:47.244 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-remoto-1.2.1-1.el8.noarch.rpm 492 kB/s | 49 kB 00:00 2023-12-01T21:58:47.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2023-12-01T21:58:47.283 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 14 MB/s | 538 kB 00:00 2023-12-01T21:58:47.343 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-babel-2.5.1-5.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2023-12-01T21:58:47.362 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): ceph-mgr-cephadm-16.2.14-328.g0dc074ce.e 374 kB/s | 116 kB 00:00 2023-12-01T21:58:47.362 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:47.363 INFO:teuthology.orchestra.run.smithi039.stdout:Total 8.9 MB/s | 5.8 MB 00:00 2023-12-01T21:58:47.367 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:58:47.377 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:58:47.377 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:58:47.479 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:58:47.484 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:58:47.642 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:48.469 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:44 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:58:49.045 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/6 2023-12-01T21:58:49.294 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/6 2023-12-01T21:58:49.624 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2023-12-01T21:58:49.944 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2023-12-01T21:58:50.173 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 5/6 2023-12-01T21:58:50.352 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 6/6 2023-12-01T21:58:50.635 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 6/6 2023-12-01T21:58:50.635 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 1/6 2023-12-01T21:58:50.635 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2023-12-01T21:58:50.635 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2023-12-01T21:58:50.636 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 4/6 2023-12-01T21:58:50.636 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 5/6 2023-12-01T21:58:50.884 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:58:50.884 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:58:50.884 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-01T21:58:50.884 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:58:50.884 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:58:50.884 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 911 k 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2023-12-01T21:58:50.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:50.886 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 994 k 2023-12-01T21:58:50.886 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.8 M 2023-12-01T21:58:50.886 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:58:50.936 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-12-01T21:58:51.103 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-16.2.14-328.g0dc074ce.el8.x86_ 4.1 MB/s | 911 kB 00:00 2023-12-01T21:58:51.105 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:51.105 INFO:teuthology.orchestra.run.smithi170.stdout:Total 4.4 MB/s | 994 kB 00:00 2023-12-01T21:58:51.106 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:58:51.127 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:58:51.127 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:58:51.210 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:58:51.215 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:58:51.392 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:58:51.728 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-12-01T21:58:51.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 6/6 2023-12-01T21:58:51.771 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:47 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:51.965 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 2/2 2023-12-01T21:58:52.362 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 2/2 2023-12-01T21:58:52.362 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/2 2023-12-01T21:58:53.321 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-12-01T21:58:53.321 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:49 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:53.792 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:58:53.902 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:53.902 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:58:53.902 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:16.2.14-328.g0dc074ce.el8.noarch 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.5.1-5.el8.noarch 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:53.903 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:58:54.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2023-12-01T21:58:54.409 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:55.260 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:58:55.316 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:51 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:55.355 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:55.356 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:58:55.356 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-12-01T21:58:55.356 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:55.356 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:58:55.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2023-12-01T21:58:55.880 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:58:56.663 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:52 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:58:57.218 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:58:57.219 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:58:57.219 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-12-01T21:58:57.219 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:58:57.219 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:58:57.219 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout: x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 911 k 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2023-12-01T21:58:57.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:58:57.221 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 994 k 2023-12-01T21:58:57.221 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 2.8 M 2023-12-01T21:58:57.221 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:58:57.254 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-12-01T21:58:57.593 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): ceph-fuse-16.2.14-328.g0dc074ce.el8.x86_ 2.4 MB/s | 911 kB 00:00 2023-12-01T21:58:57.595 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:57.595 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.6 MB/s | 994 kB 00:00 2023-12-01T21:58:57.597 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:58:57.620 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:58:57.620 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:58:57.712 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:58:57.717 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:58:57.893 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:58:58.461 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 131 k 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:58:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:58:58.463 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 131 k 2023-12-01T21:58:58.463 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 432 k 2023-12-01T21:58:58.463 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:58:58.722 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-16.2.14-328.g0dc074ce.el8.x86_64 505 kB/s | 131 kB 00:00 2023-12-01T21:58:58.723 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:58:58.723 INFO:teuthology.orchestra.run.smithi170.stdout:Total 502 kB/s | 131 kB 00:00 2023-12-01T21:58:58.724 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:58:58.730 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:58:58.731 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:58:58.787 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:58:58.791 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:58:58.919 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:58:59.037 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-12-01T21:58:59.099 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:58:59.366 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:58:59.844 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 2/2 2023-12-01T21:59:00.270 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 2/2 2023-12-01T21:59:00.270 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/2 2023-12-01T21:59:00.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:00.355 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:56 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-12-01T21:59:01.380 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:57 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:02.232 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:59:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:02.328 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:59:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:02.328 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:02.328 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:02.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2023-12-01T21:59:02.853 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:03.507 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:59:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:03.614 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:59:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-12-01T21:59:03.614 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:03.615 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:03.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2023-12-01T21:59:03.760 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:59 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:04.130 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:05.067 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:05.102 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:01 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:05.546 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:05.548 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:05.548 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:05.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2023-12-01T21:59:06.187 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:06.991 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 131 k 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:06.992 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T21:59:06.993 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:06.993 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 131 k 2023-12-01T21:59:06.993 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 432 k 2023-12-01T21:59:06.993 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:59:07.029 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:03 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:07.252 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-16.2.14-328.g0dc074ce.el8.x86_64 507 kB/s | 131 kB 00:00 2023-12-01T21:59:07.252 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:07.252 INFO:teuthology.orchestra.run.smithi039.stdout:Total 504 kB/s | 131 kB 00:00 2023-12-01T21:59:07.253 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:59:07.260 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:59:07.260 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:59:07.320 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:59:07.325 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:59:07.446 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:59:07.624 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:07.891 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:08.907 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 25 k 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:59:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:08.909 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 25 k 2023-12-01T21:59:08.909 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 107 k 2023-12-01T21:59:08.909 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:59:08.953 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:08.953 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:04 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:09.021 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-16.2.14-328.g0dc074ce.el8.x86_6 227 kB/s | 25 kB 00:00 2023-12-01T21:59:09.022 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:09.022 INFO:teuthology.orchestra.run.smithi170.stdout:Total 225 kB/s | 25 kB 00:00 2023-12-01T21:59:09.022 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:59:09.026 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:59:09.026 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:59:09.028 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:59:09.033 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:59:09.087 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:59:09.261 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:09.512 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:10.425 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:10.425 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:06 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:11.155 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:59:11.261 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:11.261 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:59:11.261 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:11.261 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:11.261 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:11.401 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2023-12-01T21:59:11.799 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:12.594 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:59:12.691 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:12.692 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:59:12.692 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:12.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:12.692 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:12.823 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:08 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:12.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2023-12-01T21:59:13.208 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:14.061 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:10 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:14.225 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:14.720 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:14.721 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:14.721 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:14.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2023-12-01T21:59:15.215 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:15.432 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:15.911 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:15.913 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:15.913 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:16.017 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2023-12-01T21:59:16.389 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:16.676 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:12 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:17.208 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:13 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:18.582 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 25 k 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:18.583 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:59:18.584 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:18.584 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T21:59:18.584 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:18.584 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 25 k 2023-12-01T21:59:18.584 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 107 k 2023-12-01T21:59:18.584 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:59:18.590 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:18.771 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-16.2.14-328.g0dc074ce.el8.x86_6 136 kB/s | 25 kB 00:00 2023-12-01T21:59:18.772 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:18.772 INFO:teuthology.orchestra.run.smithi039.stdout:Total 135 kB/s | 25 kB 00:00 2023-12-01T21:59:18.772 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:59:18.776 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:59:18.776 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:59:18.779 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:59:18.783 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:59:18.836 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:59:19.006 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:19.067 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:19.068 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:19.068 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:19.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2023-12-01T21:59:19.264 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:19.549 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:20.321 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:20.321 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:16 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:20.381 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:16 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:21.681 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:22.157 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:22.158 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:22.158 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:22.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2023-12-01T21:59:22.468 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:59:22.576 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:22.576 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:59:22.576 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:22.576 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:22.576 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:22.634 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:22.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2023-12-01T21:59:23.089 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:23.520 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:19 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:24.381 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:20 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:24.873 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:25.348 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:25.349 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:25.349 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:25.447 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2023-12-01T21:59:25.819 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:25.865 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:26.369 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:26.370 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:26.370 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:26.475 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2023-12-01T21:59:26.871 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:27.554 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:23 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:28.277 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:24 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:28.952 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:29.426 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:29.427 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:29.427 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:29.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2023-12-01T21:59:29.881 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:29.891 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:30.382 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:30.383 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:30.383 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:30.482 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2023-12-01T21:59:30.684 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:26 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:30.878 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:31.933 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:27 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:32.029 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:32.503 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:32.504 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-01T21:59:32.505 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:32.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2023-12-01T21:59:32.977 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:33.372 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:33.755 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:29 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:33.868 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:33.869 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:33.869 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:33.969 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2023-12-01T21:59:34.366 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:35.375 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:31 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:35.549 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:35.550 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:35.550 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-01T21:59:35.550 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:35.550 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 88 k 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 88 k 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 240 k 2023-12-01T21:59:35.551 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:59:35.684 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-16.2.14-328.g0dc074ce.el8.x86_64.rpm 665 kB/s | 88 kB 00:00 2023-12-01T21:59:35.685 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:35.685 INFO:teuthology.orchestra.run.smithi170.stdout:Total 658 kB/s | 88 kB 00:00 2023-12-01T21:59:35.685 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:59:35.697 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:59:35.697 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:59:35.752 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:59:35.756 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:59:35.866 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:59:36.082 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:36.350 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:36.763 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:37.262 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:37.263 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:37.264 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:37.303 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:37.304 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:33 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:37.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2023-12-01T21:59:37.758 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:38.679 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:34 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:39.148 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:59:39.244 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:39.244 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:59:39.244 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:39.244 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:39.244 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:39.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2023-12-01T21:59:39.725 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:40.138 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:40.603 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:36 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:40.636 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:40.637 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:40.637 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:40.739 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2023-12-01T21:59:41.142 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:42.049 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:38 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:42.379 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 3.7 M 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:42.380 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.7 M 2023-12-01T21:59:42.381 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 15 M 2023-12-01T21:59:42.381 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:59:43.083 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-16.2.14-328.g0dc074ce.el8.x86_64.rpm 5.2 MB/s | 3.7 MB 00:00 2023-12-01T21:59:43.084 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:43.084 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.2 MB/s | 3.7 MB 00:00 2023-12-01T21:59:43.084 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:59:43.098 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:59:43.098 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:59:43.167 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:59:43.172 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:59:43.454 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:16.2.14-328.g0dc074ce.el8.x86_64 is already installed. 2023-12-01T21:59:43.613 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:59:43.849 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:43.949 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:43.950 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-12-01T21:59:43.950 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:44.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2023-12-01T21:59:44.251 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:44.450 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:45.181 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:45.181 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:41 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:45.396 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:41 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:47.049 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:59:47.144 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:47.144 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:59:47.144 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:47.145 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:47.145 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:47.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2023-12-01T21:59:47.406 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 88 k 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T21:59:47.407 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:47.408 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 88 k 2023-12-01T21:59:47.408 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 240 k 2023-12-01T21:59:47.408 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:59:47.630 INFO:teuthology.orchestra.run.smithi170.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:47.672 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-16.2.14-328.g0dc074ce.el8.x86_64.rpm 335 kB/s | 88 kB 00:00 2023-12-01T21:59:47.673 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:47.673 INFO:teuthology.orchestra.run.smithi039.stdout:Total 333 kB/s | 88 kB 00:00 2023-12-01T21:59:47.673 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:59:47.686 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:59:47.686 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:59:47.747 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:59:47.751 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:59:47.869 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:59:48.065 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:48.366 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:48.416 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:44 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:49.438 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:49.439 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:45 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:50.229 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-01T21:59:50.231 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:50.231 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-01T21:59:50.231 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:50.231 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-01T21:59:50.231 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 175 k 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 175 k 2023-12-01T21:59:50.232 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 523 k 2023-12-01T21:59:50.233 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-01T21:59:50.383 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-16.2.14-328.g0dc074ce.el8.x86_64.rpm 1.1 MB/s | 175 kB 00:00 2023-12-01T21:59:50.384 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:50.384 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.1 MB/s | 175 kB 00:00 2023-12-01T21:59:50.384 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-01T21:59:50.396 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-01T21:59:50.396 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-01T21:59:50.452 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-01T21:59:50.456 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-01T21:59:50.582 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-01T21:59:50.804 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:51.064 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:51.445 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T21:59:51.551 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:51.551 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T21:59:51.551 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:51.551 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:51.551 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T21:59:51.675 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2023-12-01T21:59:51.996 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:51.996 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:47 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:52.072 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T21:59:52.977 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:48 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:53.930 INFO:teuthology.orchestra.run.smithi170.stdout:Installed products updated. 2023-12-01T21:59:54.027 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:54.027 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-01T21:59:54.027 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T21:59:54.028 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T21:59:54.028 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-01T21:59:54.141 DEBUG:teuthology.parallel:result is None 2023-12-01T21:59:54.948 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 3.7 M 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T21:59:54.949 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.7 M 2023-12-01T21:59:54.950 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 15 M 2023-12-01T21:59:54.950 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T21:59:55.438 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-16.2.14-328.g0dc074ce.el8.x86_64.rpm 7.5 MB/s | 3.7 MB 00:00 2023-12-01T21:59:55.439 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T21:59:55.440 INFO:teuthology.orchestra.run.smithi039.stdout:Total 7.5 MB/s | 3.7 MB 00:00 2023-12-01T21:59:55.440 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T21:59:55.455 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T21:59:55.455 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T21:59:55.529 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T21:59:55.534 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T21:59:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T21:59:56.210 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:56.640 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:57.728 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T21:59:57.729 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:53 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T21:59:59.939 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T22:00:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:00.048 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T22:00:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T22:00:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:00.048 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T22:00:00.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2023-12-01T22:00:00.567 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-12-01T22:00:01.517 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:57 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T22:00:03.463 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:16.2.14-328.g0dc074ce.el8 ceph 175 k 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 175 k 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 523 k 2023-12-01T22:00:03.464 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-12-01T22:00:03.617 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-16.2.14-328.g0dc074ce.el8.x86_64.rpm 1.1 MB/s | 175 kB 00:00 2023-12-01T22:00:03.617 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-12-01T22:00:03.618 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.1 MB/s | 175 kB 00:00 2023-12-01T22:00:03.618 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-12-01T22:00:03.632 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-12-01T22:00:03.632 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-12-01T22:00:03.692 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-12-01T22:00:03.696 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-12-01T22:00:03.817 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-12-01T22:00:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T22:00:04.293 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T22:00:05.430 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 1/1 2023-12-01T22:00:05.431 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:04:01 ago on Fri 01 Dec 2023 09:56:04 PM UTC. 2023-12-01T22:00:07.416 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-12-01T22:00:07.522 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:07.522 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-12-01T22:00:07.522 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:16.2.14-328.g0dc074ce.el8.x86_64 2023-12-01T22:00:07.522 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:07.522 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-12-01T22:00:07.664 DEBUG:teuthology.parallel:result is None 2023-12-01T22:00:07.665 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:00:07.825 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-01T22:00:07.845 INFO:teuthology.orchestra.run.smithi039.stdout:16.2.14-328.g0dc074ce.el8 2023-12-01T22:00:07.846 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-328.g0dc074ce.el8 2023-12-01T22:00:07.846 INFO:teuthology.task.install:The correct ceph version 16.2.14-328.g0dc074ce is installed. 2023-12-01T22:00:07.846 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:00:07.987 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-01T22:00:08.010 INFO:teuthology.orchestra.run.smithi170.stdout:16.2.14-328.g0dc074ce.el8 2023-12-01T22:00:08.010 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-328.g0dc074ce.el8 2023-12-01T22:00:08.010 INFO:teuthology.task.install:The correct ceph version 16.2.14-328.g0dc074ce is installed. 2023-12-01T22:00:08.013 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-01T22:00:08.013 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:08.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-01T22:00:08.041 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:08.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-01T22:00:08.078 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-01T22:00:08.079 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:08.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2023-12-01T22:00:08.106 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-01T22:00:08.171 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:08.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-12-01T22:00:08.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-01T22:00:08.263 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-01T22:00:08.263 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:08.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-01T22:00:08.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-01T22:00:08.353 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:08.353 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-01T22:00:08.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-01T22:00:08.446 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-01T22:00:08.446 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:08.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2023-12-01T22:00:08.472 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-01T22:00:08.537 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:08.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-12-01T22:00:08.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-01T22:00:08.626 INFO:teuthology.run_tasks:Running task ceph... 2023-12-01T22:00:08.697 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-01T22:00:08.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2023-12-01T22:00:08.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2023-12-01T22:00:08.723 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-01T22:00:08.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-01T22:00:08.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-01T22:00:08.788 INFO:tasks.ceph:Creating extra log directories... 2023-12-01T22:00:08.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-01T22:00:08.832 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-01T22:00:08.871 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-01T22:00:08.871 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-01T22:00:08.872 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306', 'branch': 'wip-yuri8-testing-2023-11-29-0706-pacific', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits}', 'email': None, 'first_in_suite': False, 'job_id': '7474306', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2023-11-29-0706-pacific'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri8-testing-2023-11-29-0706-pacific', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 8528, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri8-testing-2023-11-29-0706-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'targets': {'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHXlX9zAwNcMwQQU3DB/EAeYCGpSjW2Nux9nhT2vXGdTFPf5SeLso4wUcOT9/x/mg0OnwCREdKDZcPGAiTVcXiY=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWQdVma+lyjNWkZyjNVt3uVkytJ8p1wyCBD+94MlywamkEMtBb61G4EUuRHQxCXZHEFEPB8KciAtaYAKz6p/mk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-01_16:53:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-01T22:00:08.872 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-01T22:00:08.897 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-01T22:00:08.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2023-12-01T22:00:08.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-12-01T22:00:08.975 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:08.976 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2023-12-01T22:00:09.030 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-01T22:00:09.033 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 212852 Links: 1 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-12-01 21:59:56.397356177 +0000 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-12-01 21:55:17.453361699 +0000 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-12-01 21:55:17.453361699 +0000 2023-12-01T22:00:09.086 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-12-01T22:00:09.087 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-01T22:00:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-12-01T22:00:09.151 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-12-01T22:00:09.151 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000401662 s, 1.3 MB/s 2023-12-01T22:00:09.152 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-01T22:00:09.208 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2023-12-01T22:00:09.263 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-01T22:00:09.263 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:09.263 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 216994 Links: 1 2023-12-01T22:00:09.264 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:09.264 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:09.264 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-12-01 21:59:56.398356157 +0000 2023-12-01T22:00:09.264 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-12-01 21:55:17.936352066 +0000 2023-12-01T22:00:09.264 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-12-01 21:55:17.936352066 +0000 2023-12-01T22:00:09.264 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-12-01T22:00:09.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-01T22:00:09.327 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-12-01T22:00:09.327 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-12-01T22:00:09.327 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000256005 s, 2.0 MB/s 2023-12-01T22:00:09.328 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-01T22:00:09.384 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 217778 Links: 1 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-12-01 21:59:56.398356157 +0000 2023-12-01T22:00:09.442 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-12-01 21:55:18.481341197 +0000 2023-12-01T22:00:09.443 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-12-01 21:55:18.481341197 +0000 2023-12-01T22:00:09.443 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-12-01T22:00:09.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-01T22:00:09.506 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-12-01T22:00:09.506 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-12-01T22:00:09.506 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000372096 s, 1.4 MB/s 2023-12-01T22:00:09.507 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-01T22:00:09.563 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 217874 Links: 1 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-12-01 21:59:56.398356157 +0000 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-12-01 21:55:18.988331085 +0000 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-12-01 21:55:18.988331085 +0000 2023-12-01T22:00:09.619 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-12-01T22:00:09.619 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-01T22:00:09.683 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-12-01T22:00:09.683 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-12-01T22:00:09.683 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000210745 s, 2.4 MB/s 2023-12-01T22:00:09.684 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-01T22:00:09.740 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:09.740 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:09.740 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2023-12-01T22:00:09.756 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-01T22:00:09.756 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 220238 Links: 1 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-01 21:59:44.016938836 +0000 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-01 21:54:59.657124658 +0000 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-01 21:54:59.657124658 +0000 2023-12-01T22:00:09.813 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-12-01T22:00:09.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-01T22:00:09.878 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-01T22:00:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-01T22:00:09.879 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000140698 s, 3.6 MB/s 2023-12-01T22:00:09.880 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-01T22:00:09.937 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2023-12-01T22:00:09.993 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-01T22:00:09.993 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:09.993 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 221365 Links: 1 2023-12-01T22:00:09.993 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:09.993 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:09.993 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-01 21:59:44.016938836 +0000 2023-12-01T22:00:09.994 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-01 21:55:00.149113455 +0000 2023-12-01T22:00:09.994 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-01 21:55:00.149113455 +0000 2023-12-01T22:00:09.994 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-12-01T22:00:09.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-01T22:00:10.056 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-01T22:00:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-01T22:00:10.057 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000120438 s, 4.3 MB/s 2023-12-01T22:00:10.058 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-01T22:00:10.114 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 219854 Links: 1 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-01 21:59:44.016938836 +0000 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-01 21:55:00.633102433 +0000 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-01 21:55:00.633102433 +0000 2023-12-01T22:00:10.169 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-12-01T22:00:10.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-01T22:00:10.232 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-01T22:00:10.232 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-01T22:00:10.232 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000116917 s, 4.4 MB/s 2023-12-01T22:00:10.233 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-01T22:00:10.289 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2023-12-01T22:00:10.345 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-01T22:00:10.345 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T22:00:10.345 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 6h/6d Inode: 224292 Links: 1 2023-12-01T22:00:10.345 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T22:00:10.345 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T22:00:10.345 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-12-01 21:59:44.016938836 +0000 2023-12-01T22:00:10.346 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-12-01 21:55:01.125091229 +0000 2023-12-01T22:00:10.346 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-12-01 21:55:01.125091229 +0000 2023-12-01T22:00:10.346 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-12-01T22:00:10.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-01T22:00:10.408 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-12-01T22:00:10.408 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-12-01T22:00:10.408 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000134083 s, 3.8 MB/s 2023-12-01T22:00:10.409 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-01T22:00:10.465 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:10.465 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-12-01T22:00:10.465 INFO:tasks.ceph:Generating config... 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] ms inject delay max = 1 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] ms inject delay type = mon 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2023-12-01T22:00:10.466 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-12-01T22:00:10.467 INFO:tasks.ceph:[global] ms type = async 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mgr] debug monc = 10 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-01T22:00:10.467 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-12-01T22:00:10.467 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-12-01T22:00:10.468 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-01T22:00:10.469 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-01T22:00:10.469 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-01T22:00:10.469 INFO:tasks.ceph:Setting up mon.a... 2023-12-01T22:00:10.469 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-01T22:00:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.keyring 2023-12-01T22:00:10.535 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-01T22:00:10.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-01T22:00:10.642 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.39'), ('mon.c', '[v2:172.21.15.39:3301,v1:172.21.15.39:6790]'), ('mon.b', '172.21.15.170')] 2023-12-01T22:00:10.643 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.170', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-01T22:00:10.644 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:10.644 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-01T22:00:10.698 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.39 --addv c '[v2:172.21.15.39:3301,v1:172.21.15.39:6790]' --add b 172.21.15.170 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:10.780 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: generated fsid ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 0 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:fsid ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:last_changed 2023-12-01T22:00:10.779807+0000 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:created 2023-12-01T22:00:10.779807+0000 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:min_mon_release 0 (unknown) 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:election_strategy: 1 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:1: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:2: [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] mon.c 2023-12-01T22:00:10.781 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-01T22:00:10.782 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-01T22:00:10.837 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e... 2023-12-01T22:00:10.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-01T22:00:10.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-01T22:00:10.921 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2023-12-01T22:00:10.921 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.922 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2023-12-01T22:00:10.923 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.170" 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon client directed command retry = 5 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: mon election default strategy = 1 2023-12-01T22:00:10.924 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject delay max = 1 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject delay probability = 0.005 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject delay type = mon 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject internal delays = 0.002 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject socket failures = 2500 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms type = async 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2023-12-01T22:00:10.925 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2023-12-01T22:00:10.926 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 20 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 20 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 10 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2023-12-01T22:00:10.927 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify cached snaps = True 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify missing on start = True 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd memory target = 1610612736 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue = debug_random 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue cut off = debug_random 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2023-12-01T22:00:10.928 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 10 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-01T22:00:10.929 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon min osdmap epochs = 25 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune interval = 2 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune min = 15 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune txsize = 2 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub inject crc mismatch = 0.01 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub inject missing keys = 0.05 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub interval = 300 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: paxos service trim min = 5 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:10.930 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2023-12-01T22:00:10.931 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-12-01T22:00:10.932 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-12-01T22:00:10.933 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-12-01T22:00:10.934 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = "172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.170" 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: mon election default strategy = 1 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject delay max = 1 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject delay probability = 0.005 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject delay type = mon 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject internal delays = 0.002 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 2500 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: ms type = async 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 600 2023-12-01T22:00:10.935 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-12-01T22:00:10.936 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: bdev async discard = True 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: bdev enable discard = True 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 20 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 20 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 10 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2023-12-01T22:00:10.937 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify cached snaps = True 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify missing on start = True 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd memory target = 1610612736 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue = debug_random 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue cut off = debug_random 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-12-01T22:00:10.938 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 10 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-12-01T22:00:10.939 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon min osdmap epochs = 25 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune interval = 2 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune min = 15 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune txsize = 2 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon scrub inject crc mismatch = 0.01 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon scrub inject missing keys = 0.05 2023-12-01T22:00:10.940 INFO:teuthology.orchestra.run.smithi170.stdout: mon scrub interval = 300 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: paxos service trim min = 5 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.c] 2023-12-01T22:00:10.941 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.b] 2023-12-01T22:00:10.942 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-01T22:00:10.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-01T22:00:11.033 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-01T22:00:11.033 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:11.033 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-01T22:00:11.048 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:11.048 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-01T22:00:11.103 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:00:11.103 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:11.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-01T22:00:11.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-01T22:00:11.179 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:11.179 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:11.234 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:00:11.234 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:11.237 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-01T22:00:11.237 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-01T22:00:11.272 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:11.272 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:11.327 INFO:tasks.ceph:Setting up mon nodes... 2023-12-01T22:00:11.327 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-01T22:00:11.327 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-01T22:00:11.448 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-01T22:00:11.450 INFO:tasks.ceph:Setting up mds nodes... 2023-12-01T22:00:11.450 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-01T22:00:11.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-01T22:00:11.487 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-01T22:00:11.500 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-01T22:00:11.500 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-12-01T22:00:11.500 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-01T22:00:11.525 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:11.525 INFO:tasks.ceph:role: osd.0 2023-12-01T22:00:11.525 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:00:11.525 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T22:00:11.618 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-01T22:00:11.619 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T22:00:11.619 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T22:00:12.915 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-12-01T22:00:12.917 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-12-01T22:00:12.917 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-01T22:00:12.962 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-01T22:00:12.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-01T22:00:13.059 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:13.060 INFO:tasks.ceph:role: osd.1 2023-12-01T22:00:13.060 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:00:13.060 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T22:00:13.154 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T22:00:14.466 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-12-01T22:00:14.468 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-12-01T22:00:14.468 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-01T22:00:14.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-01T22:00:14.584 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-01T22:00:14.651 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:14.651 INFO:tasks.ceph:role: osd.2 2023-12-01T22:00:14.651 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:00:14.651 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-01T22:00:14.745 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T22:00:14.745 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T22:00:14.745 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T22:00:14.745 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-12-01T22:00:14.746 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T22:00:14.746 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-12-01T22:00:14.746 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T22:00:14.746 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-01T22:00:14.746 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T22:00:14.746 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T22:00:16.050 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-12-01T22:00:16.052 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-12-01T22:00:16.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-01T22:00:16.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-01T22:00:16.130 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:16.226 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:16.223+0000 7f3856b49380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-01T22:00:16.226 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:16.223+0000 7f3856b49380 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-01T22:00:16.226 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:16.223+0000 7f3856b49380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-01T22:00:16.226 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:16.223+0000 7f3856b49380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-01T22:00:16.226 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:16.223+0000 7f3856b49380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-01T22:00:16.227 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:16.224+0000 7f3856b49380 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-01T22:00:19.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-01T22:00:19.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:19.193 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:19.190+0000 7f0fe0b78380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-01T22:00:19.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:19.191+0000 7f0fe0b78380 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-01T22:00:19.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:19.191+0000 7f0fe0b78380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-01T22:00:19.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:19.191+0000 7f0fe0b78380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-01T22:00:19.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:19.191+0000 7f0fe0b78380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-01T22:00:19.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:19.191+0000 7f0fe0b78380 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-01T22:00:22.050 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-01T22:00:22.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:22.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:22.159+0000 7f591cc93380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-01T22:00:22.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:22.159+0000 7f591cc93380 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-01T22:00:22.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:22.159+0000 7f591cc93380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-01T22:00:22.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:22.159+0000 7f591cc93380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-01T22:00:22.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:22.159+0000 7f591cc93380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-01T22:00:22.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:00:22.160+0000 7f591cc93380 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-01T22:00:25.019 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-01T22:00:25.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-01T22:00:25.121 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:25.121 INFO:tasks.ceph:role: osd.3 2023-12-01T22:00:25.121 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:00:25.121 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-01T22:00:25.189 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T22:00:25.189 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T22:00:25.190 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T22:00:25.215 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-12-01T22:00:25.216 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-12-01T22:00:25.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-12-01T22:00:25.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-01T22:00:25.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-01T22:00:25.431 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:25.431 INFO:tasks.ceph:role: osd.4 2023-12-01T22:00:25.431 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:00:25.431 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-01T22:00:25.495 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T22:00:25.495 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T22:00:25.495 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T22:00:25.495 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 2023-12-01T22:00:25.495 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T22:00:25.496 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-12-01T22:00:25.496 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T22:00:25.496 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-01T22:00:25.496 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T22:00:25.496 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T22:00:25.517 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-12-01T22:00:25.519 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-12-01T22:00:25.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-12-01T22:00:25.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-01T22:00:25.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-01T22:00:25.730 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-12-01T22:00:25.730 INFO:tasks.ceph:role: osd.5 2023-12-01T22:00:25.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:00:25.731 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T22:00:25.794 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T22:00:25.816 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-12-01T22:00:25.818 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-12-01T22:00:25.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-12-01T22:00:25.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-01T22:00:25.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:26.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:26.055+0000 7ff2b2e52380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-01T22:00:26.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:26.056+0000 7ff2b2e52380 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-01T22:00:26.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:26.056+0000 7ff2b2e52380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-01T22:00:26.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:26.056+0000 7ff2b2e52380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-01T22:00:26.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:26.056+0000 7ff2b2e52380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-01T22:00:26.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:26.056+0000 7ff2b2e52380 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-01T22:00:28.903 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-01T22:00:28.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:29.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:29.007+0000 7f28e8a1b380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-01T22:00:29.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:29.008+0000 7f28e8a1b380 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-01T22:00:29.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:29.008+0000 7f28e8a1b380 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-01T22:00:29.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:29.008+0000 7f28e8a1b380 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-01T22:00:29.011 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:29.008+0000 7f28e8a1b380 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-01T22:00:29.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:29.008+0000 7f28e8a1b380 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-01T22:00:31.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-01T22:00:31.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:31.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:31.990+0000 7f28b9c43380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-01T22:00:31.993 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:31.991+0000 7f28b9c43380 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-01T22:00:31.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:31.991+0000 7f28b9c43380 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-01T22:00:31.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:31.991+0000 7f28b9c43380 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-01T22:00:31.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:31.991+0000 7f28b9c43380 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-01T22:00:31.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-01T22:00:31.991+0000 7f28b9c43380 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-01T22:00:34.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-01T22:00:34.907 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-01T22:00:34.907 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:34.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-01T22:00:34.940 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:34.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-01T22:00:35.009 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:35.009 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-01T22:00:35.078 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:35.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-01T22:00:35.105 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:35.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-01T22:00:35.171 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:35.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-01T22:00:35.237 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:35.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-01T22:00:35.303 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:35.303 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-01T22:00:35.358 INFO:tasks.ceph:Adding keys to all mons... 2023-12-01T22:00:35.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-01T22:00:35.361 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-01T22:00:35.401 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2023-12-01T22:00:35.401 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBwV2plXp1pDRAA2LL3V4sVfsdDiy7wgpr3bQ== 2023-12-01T22:00:35.401 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2023-12-01T22:00:35.401 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBzV2pl0kZ4CxAA0Onr/yCodIKe5MBLu0JSaw== 2023-12-01T22:00:35.401 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2023-12-01T22:00:35.401 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB2V2pltwKcCRAAiHWXp67qyp8F3Lb7MOXP4Q== 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBrV2pluwmpGhAAGuQC7nsFb4CQ4FtfSc+oLQ== 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB6V2plIbZdAxAAaCBWuEkx45irU94JybPdOg== 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB9V2plB4CRABAAHqEwwXpHnVa9erAhg4+TAQ== 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB/V2plcJ8nOxAAVQ+hu/mlP+OmTKMp8D0HzQ== 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2023-12-01T22:00:35.402 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBrV2pltNsAHRAAz1YIMAtmr1HMf5Je4RBVhA== 2023-12-01T22:00:35.420 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2023-12-01T22:00:35.420 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBwV2plXp1pDRAA2LL3V4sVfsdDiy7wgpr3bQ== 2023-12-01T22:00:35.420 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2023-12-01T22:00:35.420 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBzV2pl0kZ4CxAA0Onr/yCodIKe5MBLu0JSaw== 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB2V2pltwKcCRAAiHWXp67qyp8F3Lb7MOXP4Q== 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBrV2pluwmpGhAAGuQC7nsFb4CQ4FtfSc+oLQ== 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB6V2plIbZdAxAAaCBWuEkx45irU94JybPdOg== 2023-12-01T22:00:35.421 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.4] 2023-12-01T22:00:35.422 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB9V2plB4CRABAAHqEwwXpHnVa9erAhg4+TAQ== 2023-12-01T22:00:35.422 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.5] 2023-12-01T22:00:35.422 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQB/V2plcJ8nOxAAVQ+hu/mlP+OmTKMp8D0HzQ== 2023-12-01T22:00:35.422 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2023-12-01T22:00:35.422 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBrV2pltNsAHRAAz1YIMAtmr1HMf5Je4RBVhA== 2023-12-01T22:00:35.423 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.446 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.508 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.562 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-01T22:00:35.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-01T22:00:35.657 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.708 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.754 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T22:00:35.802 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-01T22:00:35.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-01T22:00:35.852 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-01T22:00:35.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-01T22:00:35.877 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-01T22:00:35.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-01T22:00:35.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-01T22:00:36.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-01T22:00:36.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-01T22:00:36.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-01T22:00:36.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-01T22:00:36.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-01T22:00:36.325 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:36.328 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-01T22:00:36.381 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-01T22:00:36.381 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-01T22:00:36.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-01T22:00:36.383 INFO:tasks.ceph.mon.a:Started 2023-12-01T22:00:36.383 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-01T22:00:36.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-01T22:00:36.385 INFO:tasks.ceph.mon.c:Started 2023-12-01T22:00:36.385 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-01T22:00:36.385 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-01T22:00:36.423 INFO:tasks.ceph.mon.b:Started 2023-12-01T22:00:36.423 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-01T22:00:36.424 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-01T22:00:36.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-01T22:00:36.426 INFO:tasks.ceph.mgr.x:Started 2023-12-01T22:00:36.426 INFO:tasks.ceph:Setting crush tunables to default 2023-12-01T22:00:36.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd crush tunables default 2023-12-01T22:00:42.636 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:42.633+0000 7f5c70805680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-01T22:00:42.644 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2023-12-01T22:00:42.656 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-01T22:00:42.656 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:42.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-01T22:00:42.685 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:42.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-01T22:00:42.706 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:42.703+0000 7f5c70805680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-01T22:00:42.751 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-12-01T22:00:42.751 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-01T22:00:42.816 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:42.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-01T22:00:42.841 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:42.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-01T22:00:42.905 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:00:42.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-01T22:00:42.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 2c2da89b-1a10-4976-9a09-37fa58791d03 0 2023-12-01T22:00:43.223 INFO:teuthology.orchestra.run.smithi170.stdout:0 2023-12-01T22:00:43.224 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:43.222+0000 7f5c70805680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-01T22:00:43.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new b735f840-37fe-4e12-b352-4bcdd00b86fa 1 2023-12-01T22:00:43.448 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-12-01T22:00:43.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new dfb64dc2-d0c6-446b-94e5-e2863d03584a 2 2023-12-01T22:00:43.671 INFO:teuthology.orchestra.run.smithi170.stdout:2 2023-12-01T22:00:43.684 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 735a78cf-3c61-4168-b264-13fc8620d848 3 2023-12-01T22:00:43.894 INFO:teuthology.orchestra.run.smithi170.stdout:3 2023-12-01T22:00:43.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 5d42499a-5964-4b40-8e56-00a6ac63fe90 4 2023-12-01T22:00:44.012 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:44.010+0000 7f5c70805680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-01T22:00:44.084 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:44.082+0000 7f5c70805680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-01T22:00:44.145 INFO:teuthology.orchestra.run.smithi170.stdout:4 2023-12-01T22:00:44.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 09acc711-f491-4a2d-8976-79120d86045d 5 2023-12-01T22:00:44.286 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:44.283+0000 7f5c70805680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-01T22:00:44.393 INFO:teuthology.orchestra.run.smithi170.stdout:5 2023-12-01T22:00:44.406 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-01T22:00:44.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-01T22:00:44.409 INFO:tasks.ceph.osd.0:Started 2023-12-01T22:00:44.409 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-01T22:00:44.409 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-01T22:00:44.411 INFO:tasks.ceph.osd.1:Started 2023-12-01T22:00:44.412 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-01T22:00:44.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-01T22:00:44.414 INFO:tasks.ceph.osd.2:Started 2023-12-01T22:00:44.414 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-01T22:00:44.414 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-01T22:00:44.416 INFO:tasks.ceph.osd.3:Started 2023-12-01T22:00:44.416 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-01T22:00:44.416 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-01T22:00:44.418 INFO:tasks.ceph.osd.4:Started 2023-12-01T22:00:44.418 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-01T22:00:44.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-01T22:00:44.420 INFO:tasks.ceph.osd.5:Started 2023-12-01T22:00:44.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T22:00:44.636 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:44.637 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":8,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:00:44.380716+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T22:00:44.380716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:00:44.649 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-01T22:00:44.650 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-01T22:00:44.749 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:44.747+0000 7f5c70805680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-01T22:00:44.854 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:44.851+0000 7f5c70805680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-01T22:00:45.009 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:45.006+0000 7f5c70805680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-01T22:00:45.011 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-12-01T22:00:45.008+0000 7f35d25e7380 -1 Falling back to public interface 2023-12-01T22:00:45.012 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-12-01T22:00:45.010+0000 7ff234256380 -1 Falling back to public interface 2023-12-01T22:00:45.019 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-12-01T22:00:45.016+0000 7fba13f9e380 -1 Falling back to public interface 2023-12-01T22:00:45.023 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-12-01T22:00:45.022+0000 7f83401c7380 -1 Falling back to public interface 2023-12-01T22:00:45.025 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-12-01T22:00:45.024+0000 7f699edc0380 -1 Falling back to public interface 2023-12-01T22:00:45.027 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-12-01T22:00:45.025+0000 7fed344bf380 -1 Falling back to public interface 2023-12-01T22:00:45.087 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:45.084+0000 7f5c70805680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-01T22:00:45.155 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:45.153+0000 7f5c70805680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-01T22:00:45.253 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T22:00:45.420 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:45.417+0000 7f5c70805680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-01T22:00:45.617 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:45.614+0000 7f5c70805680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-01T22:00:46.279 INFO:teuthology.misc.health.smithi039.stdout: 2023-12-01T22:00:46.279 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":8,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:00:44.380716+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T22:00:44.380716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:00:46.284 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.282+0000 7f5c70805680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-01T22:00:46.294 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-12-01T22:00:46.354 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.351+0000 7f5c70805680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-01T22:00:46.434 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.432+0000 7f5c70805680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-01T22:00:46.582 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.580+0000 7f5c70805680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-01T22:00:46.654 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.652+0000 7f5c70805680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-01T22:00:46.772 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.770+0000 7f5c70805680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-01T22:00:46.879 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:46.876+0000 7f5c70805680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-01T22:00:47.206 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:47.205+0000 7f5c70805680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-01T22:00:47.277 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:47.276+0000 7f5c70805680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-01T22:00:48.332 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:48.331+0000 7f5c26a27700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-01T22:00:48.332 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-12-01T22:00:48.331+0000 7f5c26a27700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-01T22:00:48.476 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-12-01T22:00:48.474+0000 7f83401c7380 -1 osd.2 0 log_to_monitors {default=true} 2023-12-01T22:00:48.494 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-12-01T22:00:48.492+0000 7f35d25e7380 -1 osd.3 0 log_to_monitors {default=true} 2023-12-01T22:00:48.504 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-12-01T22:00:48.503+0000 7fed344bf380 -1 osd.0 0 log_to_monitors {default=true} 2023-12-01T22:00:48.508 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-12-01T22:00:48.507+0000 7f699edc0380 -1 osd.1 0 log_to_monitors {default=true} 2023-12-01T22:00:48.531 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-12-01T22:00:48.530+0000 7ff234256380 -1 osd.4 0 log_to_monitors {default=true} 2023-12-01T22:00:48.532 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-12-01T22:00:48.531+0000 7fba13f9e380 -1 osd.5 0 log_to_monitors {default=true} 2023-12-01T22:00:50.343 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-12-01T22:00:50.342+0000 7fed2ae7c700 -1 osd.0 0 waiting for initial osdmap 2023-12-01T22:00:50.344 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-12-01T22:00:50.342+0000 7f8336b84700 -1 osd.2 0 waiting for initial osdmap 2023-12-01T22:00:50.344 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-12-01T22:00:50.342+0000 7f35ca7a7700 -1 osd.3 0 waiting for initial osdmap 2023-12-01T22:00:50.345 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-12-01T22:00:50.342+0000 7fba0a95b700 -1 osd.5 0 waiting for initial osdmap 2023-12-01T22:00:50.345 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-12-01T22:00:50.342+0000 7ff22ac13700 -1 osd.4 0 waiting for initial osdmap 2023-12-01T22:00:50.346 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-12-01T22:00:50.343+0000 7f699577d700 -1 osd.1 0 waiting for initial osdmap 2023-12-01T22:00:50.356 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-12-01T22:00:50.355+0000 7fed265f6700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:50.356 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-12-01T22:00:50.355+0000 7f6990ef7700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:50.359 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-12-01T22:00:50.358+0000 7f83322fe700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:50.361 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-12-01T22:00:50.359+0000 7f35c471e700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:50.361 INFO:tasks.ceph.osd.4.smithi170.stderr:2023-12-01T22:00:50.359+0000 7ff22638d700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:50.362 INFO:tasks.ceph.osd.5.smithi170.stderr:2023-12-01T22:00:50.360+0000 7fba060d5700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:50.375 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-12-01T22:00:50.374+0000 7fed1bc70700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T22:00:52.899 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T22:00:53.216 INFO:teuthology.misc.health.smithi039.stdout: 2023-12-01T22:00:53.216 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":12,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:00:52.351745+0000","last_up_change":"2023-12-01T22:00:51.347488+0000","last_in_change":"2023-12-01T22:00:44.380716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-01T22:00:51.401944+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":86687},{"type":"v1","addr":"172.21.15.39:6815","nonce":86687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6817","nonce":86687},{"type":"v1","addr":"172.21.15.39:6819","nonce":86687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":86687},{"type":"v1","addr":"172.21.15.39:6823","nonce":86687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":86687},{"type":"v1","addr":"172.21.15.39:6821","nonce":86687}]},"public_addr":"172.21.15.39:6815/86687","cluster_addr":"172.21.15.39:6819/86687","heartbeat_back_addr":"172.21.15.39:6823/86687","heartbeat_front_addr":"172.21.15.39:6821/86687","state":["exists","up"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":86688},{"type":"v1","addr":"172.21.15.39:6809","nonce":86688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":86688},{"type":"v1","addr":"172.21.15.39:6811","nonce":86688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":86688},{"type":"v1","addr":"172.21.15.39:6818","nonce":86688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":86688},{"type":"v1","addr":"172.21.15.39:6814","nonce":86688}]},"public_addr":"172.21.15.39:6809/86688","cluster_addr":"172.21.15.39:6811/86688","heartbeat_back_addr":"172.21.15.39:6818/86688","heartbeat_front_addr":"172.21.15.39:6814/86688","state":["exists","up"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":86689},{"type":"v1","addr":"172.21.15.39:6801","nonce":86689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":86689},{"type":"v1","addr":"172.21.15.39:6803","nonce":86689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":86689},{"type":"v1","addr":"172.21.15.39:6807","nonce":86689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":86689},{"type":"v1","addr":"172.21.15.39:6805","nonce":86689}]},"public_addr":"172.21.15.39:6801/86689","cluster_addr":"172.21.15.39:6803/86689","heartbeat_back_addr":"172.21.15.39:6807/86689","heartbeat_front_addr":"172.21.15.39:6805/86689","state":["exists","up"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86579},{"type":"v1","addr":"172.21.15.170:6801","nonce":86579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86579},{"type":"v1","addr":"172.21.15.170:6803","nonce":86579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86579},{"type":"v1","addr":"172.21.15.170:6807","nonce":86579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86579},{"type":"v1","addr":"172.21.15.170:6805","nonce":86579}]},"public_addr":"172.21.15.170:6801/86579","cluster_addr":"172.21.15.170:6803/86579","heartbeat_back_addr":"172.21.15.170:6807/86579","heartbeat_front_addr":"172.21.15.170:6805/86579","state":["exists","up"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86580},{"type":"v1","addr":"172.21.15.170:6809","nonce":86580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86580},{"type":"v1","addr":"172.21.15.170:6811","nonce":86580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86580},{"type":"v1","addr":"172.21.15.170:6815","nonce":86580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86580},{"type":"v1","addr":"172.21.15.170:6813","nonce":86580}]},"public_addr":"172.21.15.170:6809/86580","cluster_addr":"172.21.15.170:6811/86580","heartbeat_back_addr":"172.21.15.170:6815/86580","heartbeat_front_addr":"172.21.15.170:6813/86580","state":["exists","up"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":86581},{"type":"v1","addr":"172.21.15.170:6817","nonce":86581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86581},{"type":"v1","addr":"172.21.15.170:6819","nonce":86581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86581},{"type":"v1","addr":"172.21.15.170:6823","nonce":86581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86581},{"type":"v1","addr":"172.21.15.170:6821","nonce":86581}]},"public_addr":"172.21.15.170:6817/86581","cluster_addr":"172.21.15.170:6819/86581","heartbeat_back_addr":"172.21.15.170:6823/86581","heartbeat_front_addr":"172.21.15.170:6821/86581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.528486+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.550125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.433234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.495896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.515277+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.545278+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:00:53.229 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-12-01T22:00:53.229 INFO:tasks.ceph:Creating RBD pool 2023-12-01T22:00:53.230 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-01T22:00:54.380 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'rbd' created 2023-12-01T22:00:54.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-01T22:00:55.382 INFO:teuthology.orchestra.run.smithi039.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-01T22:00:55.396 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-01T22:00:55.397 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-01T22:00:55.397 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-01T22:00:55.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-01T22:00:55.397 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-01T22:00:55.758 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:55.770 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":86148},{"type":"v1","addr":"172.21.15.170:6825","nonce":86148}]},"active_addr":"172.21.15.170:6825/86148","active_change":"2023-12-01T22:00:47.290028+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":2483310844}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":311569271}]}]} 2023-12-01T22:00:55.774 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-01T22:00:55.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-01T22:00:55.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T22:00:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:56.078 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:00:55.367234+0000","last_up_change":"2023-12-01T22:00:51.347488+0000","last_in_change":"2023-12-01T22:00:44.380716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-01T22:00:51.401944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T22:00:53.575133+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":86687},{"type":"v1","addr":"172.21.15.39:6815","nonce":86687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6817","nonce":86687},{"type":"v1","addr":"172.21.15.39:6819","nonce":86687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":86687},{"type":"v1","addr":"172.21.15.39:6823","nonce":86687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":86687},{"type":"v1","addr":"172.21.15.39:6821","nonce":86687}]},"public_addr":"172.21.15.39:6815/86687","cluster_addr":"172.21.15.39:6819/86687","heartbeat_back_addr":"172.21.15.39:6823/86687","heartbeat_front_addr":"172.21.15.39:6821/86687","state":["exists","up"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":86688},{"type":"v1","addr":"172.21.15.39:6809","nonce":86688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":86688},{"type":"v1","addr":"172.21.15.39:6811","nonce":86688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":86688},{"type":"v1","addr":"172.21.15.39:6818","nonce":86688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":86688},{"type":"v1","addr":"172.21.15.39:6814","nonce":86688}]},"public_addr":"172.21.15.39:6809/86688","cluster_addr":"172.21.15.39:6811/86688","heartbeat_back_addr":"172.21.15.39:6818/86688","heartbeat_front_addr":"172.21.15.39:6814/86688","state":["exists","up"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":86689},{"type":"v1","addr":"172.21.15.39:6801","nonce":86689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":86689},{"type":"v1","addr":"172.21.15.39:6803","nonce":86689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":86689},{"type":"v1","addr":"172.21.15.39:6807","nonce":86689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":86689},{"type":"v1","addr":"172.21.15.39:6805","nonce":86689}]},"public_addr":"172.21.15.39:6801/86689","cluster_addr":"172.21.15.39:6803/86689","heartbeat_back_addr":"172.21.15.39:6807/86689","heartbeat_front_addr":"172.21.15.39:6805/86689","state":["exists","up"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86579},{"type":"v1","addr":"172.21.15.170:6801","nonce":86579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86579},{"type":"v1","addr":"172.21.15.170:6803","nonce":86579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86579},{"type":"v1","addr":"172.21.15.170:6807","nonce":86579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86579},{"type":"v1","addr":"172.21.15.170:6805","nonce":86579}]},"public_addr":"172.21.15.170:6801/86579","cluster_addr":"172.21.15.170:6803/86579","heartbeat_back_addr":"172.21.15.170:6807/86579","heartbeat_front_addr":"172.21.15.170:6805/86579","state":["exists","up"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86580},{"type":"v1","addr":"172.21.15.170:6809","nonce":86580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86580},{"type":"v1","addr":"172.21.15.170:6811","nonce":86580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86580},{"type":"v1","addr":"172.21.15.170:6815","nonce":86580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86580},{"type":"v1","addr":"172.21.15.170:6813","nonce":86580}]},"public_addr":"172.21.15.170:6809/86580","cluster_addr":"172.21.15.170:6811/86580","heartbeat_back_addr":"172.21.15.170:6815/86580","heartbeat_front_addr":"172.21.15.170:6813/86580","state":["exists","up"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":86581},{"type":"v1","addr":"172.21.15.170:6817","nonce":86581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86581},{"type":"v1","addr":"172.21.15.170:6819","nonce":86581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86581},{"type":"v1","addr":"172.21.15.170:6823","nonce":86581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86581},{"type":"v1","addr":"172.21.15.170:6821","nonce":86581}]},"public_addr":"172.21.15.170:6817/86581","cluster_addr":"172.21.15.170:6819/86581","heartbeat_back_addr":"172.21.15.170:6823/86581","heartbeat_front_addr":"172.21.15.170:6821/86581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.528486+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.550125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.433234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.495896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.515277+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.545278+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:00:56.091 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-01T22:00:56.091 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T22:00:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:00:56.396 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":16,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:00:56.370873+0000","last_up_change":"2023-12-01T22:00:51.347488+0000","last_in_change":"2023-12-01T22:00:44.380716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-01T22:00:51.401944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T22:00:53.575133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":86687},{"type":"v1","addr":"172.21.15.39:6815","nonce":86687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6817","nonce":86687},{"type":"v1","addr":"172.21.15.39:6819","nonce":86687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":86687},{"type":"v1","addr":"172.21.15.39:6823","nonce":86687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":86687},{"type":"v1","addr":"172.21.15.39:6821","nonce":86687}]},"public_addr":"172.21.15.39:6815/86687","cluster_addr":"172.21.15.39:6819/86687","heartbeat_back_addr":"172.21.15.39:6823/86687","heartbeat_front_addr":"172.21.15.39:6821/86687","state":["exists","up"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":86688},{"type":"v1","addr":"172.21.15.39:6809","nonce":86688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":86688},{"type":"v1","addr":"172.21.15.39:6811","nonce":86688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":86688},{"type":"v1","addr":"172.21.15.39:6818","nonce":86688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":86688},{"type":"v1","addr":"172.21.15.39:6814","nonce":86688}]},"public_addr":"172.21.15.39:6809/86688","cluster_addr":"172.21.15.39:6811/86688","heartbeat_back_addr":"172.21.15.39:6818/86688","heartbeat_front_addr":"172.21.15.39:6814/86688","state":["exists","up"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":86689},{"type":"v1","addr":"172.21.15.39:6801","nonce":86689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":86689},{"type":"v1","addr":"172.21.15.39:6803","nonce":86689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":86689},{"type":"v1","addr":"172.21.15.39:6807","nonce":86689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":86689},{"type":"v1","addr":"172.21.15.39:6805","nonce":86689}]},"public_addr":"172.21.15.39:6801/86689","cluster_addr":"172.21.15.39:6803/86689","heartbeat_back_addr":"172.21.15.39:6807/86689","heartbeat_front_addr":"172.21.15.39:6805/86689","state":["exists","up"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86579},{"type":"v1","addr":"172.21.15.170:6801","nonce":86579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86579},{"type":"v1","addr":"172.21.15.170:6803","nonce":86579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86579},{"type":"v1","addr":"172.21.15.170:6807","nonce":86579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86579},{"type":"v1","addr":"172.21.15.170:6805","nonce":86579}]},"public_addr":"172.21.15.170:6801/86579","cluster_addr":"172.21.15.170:6803/86579","heartbeat_back_addr":"172.21.15.170:6807/86579","heartbeat_front_addr":"172.21.15.170:6805/86579","state":["exists","up"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86580},{"type":"v1","addr":"172.21.15.170:6809","nonce":86580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86580},{"type":"v1","addr":"172.21.15.170:6811","nonce":86580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86580},{"type":"v1","addr":"172.21.15.170:6815","nonce":86580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86580},{"type":"v1","addr":"172.21.15.170:6813","nonce":86580}]},"public_addr":"172.21.15.170:6809/86580","cluster_addr":"172.21.15.170:6811/86580","heartbeat_back_addr":"172.21.15.170:6815/86580","heartbeat_front_addr":"172.21.15.170:6813/86580","state":["exists","up"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":86581},{"type":"v1","addr":"172.21.15.170:6817","nonce":86581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86581},{"type":"v1","addr":"172.21.15.170:6819","nonce":86581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86581},{"type":"v1","addr":"172.21.15.170:6823","nonce":86581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86581},{"type":"v1","addr":"172.21.15.170:6821","nonce":86581}]},"public_addr":"172.21.15.170:6817/86581","cluster_addr":"172.21.15.170:6819/86581","heartbeat_back_addr":"172.21.15.170:6823/86581","heartbeat_front_addr":"172.21.15.170:6821/86581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.528486+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.550125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.433234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.495896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.515277+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.545278+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:00:56.410 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-01T22:00:56.573 INFO:teuthology.orchestra.run.smithi039.stdout:47244640258 2023-12-01T22:00:56.573 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-01T22:00:56.733 INFO:teuthology.orchestra.run.smithi039.stdout:47244640260 2023-12-01T22:00:56.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-01T22:00:56.892 INFO:teuthology.orchestra.run.smithi039.stdout:47244640258 2023-12-01T22:00:56.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-01T22:00:57.052 INFO:teuthology.orchestra.run.smithi039.stdout:47244640259 2023-12-01T22:00:57.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-01T22:00:57.213 INFO:teuthology.orchestra.run.smithi039.stdout:47244640259 2023-12-01T22:00:57.213 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-01T22:00:57.373 INFO:teuthology.orchestra.run.smithi039.stdout:47244640261 2023-12-01T22:00:57.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-01T22:00:57.679 INFO:teuthology.orchestra.run.smithi039.stdout:47244640258 2023-12-01T22:00:57.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.0 2023-12-01T22:00:57.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-01T22:00:57.998 INFO:teuthology.orchestra.run.smithi039.stdout:47244640261 2023-12-01T22:00:58.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.1 2023-12-01T22:00:58.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-01T22:00:58.320 INFO:teuthology.orchestra.run.smithi039.stdout:47244640258 2023-12-01T22:00:58.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.2 2023-12-01T22:00:58.334 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-01T22:00:58.641 INFO:teuthology.orchestra.run.smithi039.stdout:47244640259 2023-12-01T22:00:58.656 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2023-12-01T22:00:58.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-01T22:00:58.964 INFO:teuthology.orchestra.run.smithi039.stdout:47244640259 2023-12-01T22:00:58.978 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2023-12-01T22:00:58.978 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-01T22:00:59.287 INFO:teuthology.orchestra.run.smithi039.stdout:47244640259 2023-12-01T22:00:59.300 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.5 2023-12-01T22:01:00.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-01T22:01:00.626 INFO:teuthology.orchestra.run.smithi039.stdout:47244640261 2023-12-01T22:01:00.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.5 2023-12-01T22:01:00.639 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-01T22:01:00.640 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:01:00.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:01:00.943 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:01:00.954 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-12-01T22:00:59.300772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":269082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.436530+0000","last_change":"2023-12-01T22:00:55.435954+0000","last_active":"2023-12-01T22:00:55.436530+0000","last_peered":"2023-12-01T22:00:55.436530+0000","last_clean":"2023-12-01T22:00:55.436530+0000","last_became_active":"2023-12-01T22:00:55.435598+0000","last_became_peered":"2023-12-01T22:00:55.435598+0000","last_unstale":"2023-12-01T22:00:55.436530+0000","last_undegraded":"2023-12-01T22:00:55.436530+0000","last_fullsized":"2023-12-01T22:00:55.436530+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383085+0000","last_change":"2023-12-01T22:00:55.380208+0000","last_active":"2023-12-01T22:00:55.383085+0000","last_peered":"2023-12-01T22:00:55.383085+0000","last_clean":"2023-12-01T22:00:55.383085+0000","last_became_active":"2023-12-01T22:00:55.379870+0000","last_became_peered":"2023-12-01T22:00:55.379870+0000","last_unstale":"2023-12-01T22:00:55.383085+0000","last_undegraded":"2023-12-01T22:00:55.383085+0000","last_fullsized":"2023-12-01T22:00:55.383085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412205+0000","last_change":"2023-12-01T22:00:55.411681+0000","last_active":"2023-12-01T22:00:55.412205+0000","last_peered":"2023-12-01T22:00:55.412205+0000","last_clean":"2023-12-01T22:00:55.412205+0000","last_became_active":"2023-12-01T22:00:55.411272+0000","last_became_peered":"2023-12-01T22:00:55.411272+0000","last_unstale":"2023-12-01T22:00:55.412205+0000","last_undegraded":"2023-12-01T22:00:55.412205+0000","last_fullsized":"2023-12-01T22:00:55.412205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383697+0000","last_change":"2023-12-01T22:00:55.383115+0000","last_active":"2023-12-01T22:00:55.383697+0000","last_peered":"2023-12-01T22:00:55.383697+0000","last_clean":"2023-12-01T22:00:55.383697+0000","last_became_active":"2023-12-01T22:00:55.382740+0000","last_became_peered":"2023-12-01T22:00:55.382740+0000","last_unstale":"2023-12-01T22:00:55.383697+0000","last_undegraded":"2023-12-01T22:00:55.383697+0000","last_fullsized":"2023-12-01T22:00:55.383697+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.373530+0000","last_change":"2023-12-01T22:00:53.374514+0000","last_active":"2023-12-01T22:00:55.373530+0000","last_peered":"2023-12-01T22:00:55.373530+0000","last_clean":"2023-12-01T22:00:55.373530+0000","last_became_active":"2023-12-01T22:00:53.374199+0000","last_became_peered":"2023-12-01T22:00:53.374199+0000","last_unstale":"2023-12-01T22:00:55.373530+0000","last_undegraded":"2023-12-01T22:00:55.373530+0000","last_fullsized":"2023-12-01T22:00:55.373530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.384148+0000","last_change":"2023-12-01T22:00:55.383420+0000","last_active":"2023-12-01T22:00:55.384148+0000","last_peered":"2023-12-01T22:00:55.384148+0000","last_clean":"2023-12-01T22:00:55.384148+0000","last_became_active":"2023-12-01T22:00:55.382971+0000","last_became_peered":"2023-12-01T22:00:55.382971+0000","last_unstale":"2023-12-01T22:00:55.384148+0000","last_undegraded":"2023-12-01T22:00:55.384148+0000","last_fullsized":"2023-12-01T22:00:55.384148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.386532+0000","last_change":"2023-12-01T22:00:55.383868+0000","last_active":"2023-12-01T22:00:55.386532+0000","last_peered":"2023-12-01T22:00:55.386532+0000","last_clean":"2023-12-01T22:00:55.386532+0000","last_became_active":"2023-12-01T22:00:55.383492+0000","last_became_peered":"2023-12-01T22:00:55.383492+0000","last_unstale":"2023-12-01T22:00:55.386532+0000","last_undegraded":"2023-12-01T22:00:55.386532+0000","last_fullsized":"2023-12-01T22:00:55.386532+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383918+0000","last_change":"2023-12-01T22:00:55.383150+0000","last_active":"2023-12-01T22:00:55.383918+0000","last_peered":"2023-12-01T22:00:55.383918+0000","last_clean":"2023-12-01T22:00:55.383918+0000","last_became_active":"2023-12-01T22:00:55.382812+0000","last_became_peered":"2023-12-01T22:00:55.382812+0000","last_unstale":"2023-12-01T22:00:55.383918+0000","last_undegraded":"2023-12-01T22:00:55.383918+0000","last_fullsized":"2023-12-01T22:00:55.383918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412954+0000","last_change":"2023-12-01T22:00:55.411524+0000","last_active":"2023-12-01T22:00:55.412954+0000","last_peered":"2023-12-01T22:00:55.412954+0000","last_clean":"2023-12-01T22:00:55.412954+0000","last_became_active":"2023-12-01T22:00:55.411153+0000","last_became_peered":"2023-12-01T22:00:55.411153+0000","last_unstale":"2023-12-01T22:00:55.412954+0000","last_undegraded":"2023-12-01T22:00:55.412954+0000","last_fullsized":"2023-12-01T22:00:55.412954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:01:00.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:01:01.255 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:01:01.257 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:01:01.268 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-12-01T22:00:59.300772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":269082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.436530+0000","last_change":"2023-12-01T22:00:55.435954+0000","last_active":"2023-12-01T22:00:55.436530+0000","last_peered":"2023-12-01T22:00:55.436530+0000","last_clean":"2023-12-01T22:00:55.436530+0000","last_became_active":"2023-12-01T22:00:55.435598+0000","last_became_peered":"2023-12-01T22:00:55.435598+0000","last_unstale":"2023-12-01T22:00:55.436530+0000","last_undegraded":"2023-12-01T22:00:55.436530+0000","last_fullsized":"2023-12-01T22:00:55.436530+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383085+0000","last_change":"2023-12-01T22:00:55.380208+0000","last_active":"2023-12-01T22:00:55.383085+0000","last_peered":"2023-12-01T22:00:55.383085+0000","last_clean":"2023-12-01T22:00:55.383085+0000","last_became_active":"2023-12-01T22:00:55.379870+0000","last_became_peered":"2023-12-01T22:00:55.379870+0000","last_unstale":"2023-12-01T22:00:55.383085+0000","last_undegraded":"2023-12-01T22:00:55.383085+0000","last_fullsized":"2023-12-01T22:00:55.383085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412205+0000","last_change":"2023-12-01T22:00:55.411681+0000","last_active":"2023-12-01T22:00:55.412205+0000","last_peered":"2023-12-01T22:00:55.412205+0000","last_clean":"2023-12-01T22:00:55.412205+0000","last_became_active":"2023-12-01T22:00:55.411272+0000","last_became_peered":"2023-12-01T22:00:55.411272+0000","last_unstale":"2023-12-01T22:00:55.412205+0000","last_undegraded":"2023-12-01T22:00:55.412205+0000","last_fullsized":"2023-12-01T22:00:55.412205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383697+0000","last_change":"2023-12-01T22:00:55.383115+0000","last_active":"2023-12-01T22:00:55.383697+0000","last_peered":"2023-12-01T22:00:55.383697+0000","last_clean":"2023-12-01T22:00:55.383697+0000","last_became_active":"2023-12-01T22:00:55.382740+0000","last_became_peered":"2023-12-01T22:00:55.382740+0000","last_unstale":"2023-12-01T22:00:55.383697+0000","last_undegraded":"2023-12-01T22:00:55.383697+0000","last_fullsized":"2023-12-01T22:00:55.383697+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.373530+0000","last_change":"2023-12-01T22:00:53.374514+0000","last_active":"2023-12-01T22:00:55.373530+0000","last_peered":"2023-12-01T22:00:55.373530+0000","last_clean":"2023-12-01T22:00:55.373530+0000","last_became_active":"2023-12-01T22:00:53.374199+0000","last_became_peered":"2023-12-01T22:00:53.374199+0000","last_unstale":"2023-12-01T22:00:55.373530+0000","last_undegraded":"2023-12-01T22:00:55.373530+0000","last_fullsized":"2023-12-01T22:00:55.373530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.384148+0000","last_change":"2023-12-01T22:00:55.383420+0000","last_active":"2023-12-01T22:00:55.384148+0000","last_peered":"2023-12-01T22:00:55.384148+0000","last_clean":"2023-12-01T22:00:55.384148+0000","last_became_active":"2023-12-01T22:00:55.382971+0000","last_became_peered":"2023-12-01T22:00:55.382971+0000","last_unstale":"2023-12-01T22:00:55.384148+0000","last_undegraded":"2023-12-01T22:00:55.384148+0000","last_fullsized":"2023-12-01T22:00:55.384148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.386532+0000","last_change":"2023-12-01T22:00:55.383868+0000","last_active":"2023-12-01T22:00:55.386532+0000","last_peered":"2023-12-01T22:00:55.386532+0000","last_clean":"2023-12-01T22:00:55.386532+0000","last_became_active":"2023-12-01T22:00:55.383492+0000","last_became_peered":"2023-12-01T22:00:55.383492+0000","last_unstale":"2023-12-01T22:00:55.386532+0000","last_undegraded":"2023-12-01T22:00:55.386532+0000","last_fullsized":"2023-12-01T22:00:55.386532+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383918+0000","last_change":"2023-12-01T22:00:55.383150+0000","last_active":"2023-12-01T22:00:55.383918+0000","last_peered":"2023-12-01T22:00:55.383918+0000","last_clean":"2023-12-01T22:00:55.383918+0000","last_became_active":"2023-12-01T22:00:55.382812+0000","last_became_peered":"2023-12-01T22:00:55.382812+0000","last_unstale":"2023-12-01T22:00:55.383918+0000","last_undegraded":"2023-12-01T22:00:55.383918+0000","last_fullsized":"2023-12-01T22:00:55.383918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412954+0000","last_change":"2023-12-01T22:00:55.411524+0000","last_active":"2023-12-01T22:00:55.412954+0000","last_peered":"2023-12-01T22:00:55.412954+0000","last_clean":"2023-12-01T22:00:55.412954+0000","last_became_active":"2023-12-01T22:00:55.411153+0000","last_became_peered":"2023-12-01T22:00:55.411153+0000","last_unstale":"2023-12-01T22:00:55.412954+0000","last_undegraded":"2023-12-01T22:00:55.412954+0000","last_fullsized":"2023-12-01T22:00:55.412954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:01:01.269 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-01T22:01:01.269 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-01T22:01:01.269 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-01T22:01:01.270 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-01T22:01:01.618 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:01:01.633 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-01T22:01:01.634 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-01T22:01:01.635 INFO:teuthology.run_tasks:Running task mon_thrash... 2023-12-01T22:01:01.669 INFO:tasks.mon_thrash:Beginning mon_thrash... 2023-12-01T22:01:01.669 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T22:01:01.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:01:01.976 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":16,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:00:56.370873+0000","last_up_change":"2023-12-01T22:00:51.347488+0000","last_in_change":"2023-12-01T22:00:44.380716+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-01T22:00:51.401944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T22:00:53.575133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":86687},{"type":"v1","addr":"172.21.15.39:6815","nonce":86687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6817","nonce":86687},{"type":"v1","addr":"172.21.15.39:6819","nonce":86687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":86687},{"type":"v1","addr":"172.21.15.39:6823","nonce":86687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":86687},{"type":"v1","addr":"172.21.15.39:6821","nonce":86687}]},"public_addr":"172.21.15.39:6815/86687","cluster_addr":"172.21.15.39:6819/86687","heartbeat_back_addr":"172.21.15.39:6823/86687","heartbeat_front_addr":"172.21.15.39:6821/86687","state":["exists","up"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":86688},{"type":"v1","addr":"172.21.15.39:6809","nonce":86688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":86688},{"type":"v1","addr":"172.21.15.39:6811","nonce":86688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":86688},{"type":"v1","addr":"172.21.15.39:6818","nonce":86688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":86688},{"type":"v1","addr":"172.21.15.39:6814","nonce":86688}]},"public_addr":"172.21.15.39:6809/86688","cluster_addr":"172.21.15.39:6811/86688","heartbeat_back_addr":"172.21.15.39:6818/86688","heartbeat_front_addr":"172.21.15.39:6814/86688","state":["exists","up"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":86689},{"type":"v1","addr":"172.21.15.39:6801","nonce":86689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":86689},{"type":"v1","addr":"172.21.15.39:6803","nonce":86689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":86689},{"type":"v1","addr":"172.21.15.39:6807","nonce":86689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":86689},{"type":"v1","addr":"172.21.15.39:6805","nonce":86689}]},"public_addr":"172.21.15.39:6801/86689","cluster_addr":"172.21.15.39:6803/86689","heartbeat_back_addr":"172.21.15.39:6807/86689","heartbeat_front_addr":"172.21.15.39:6805/86689","state":["exists","up"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86579},{"type":"v1","addr":"172.21.15.170:6801","nonce":86579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86579},{"type":"v1","addr":"172.21.15.170:6803","nonce":86579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86579},{"type":"v1","addr":"172.21.15.170:6807","nonce":86579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86579},{"type":"v1","addr":"172.21.15.170:6805","nonce":86579}]},"public_addr":"172.21.15.170:6801/86579","cluster_addr":"172.21.15.170:6803/86579","heartbeat_back_addr":"172.21.15.170:6807/86579","heartbeat_front_addr":"172.21.15.170:6805/86579","state":["exists","up"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86580},{"type":"v1","addr":"172.21.15.170:6809","nonce":86580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86580},{"type":"v1","addr":"172.21.15.170:6811","nonce":86580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86580},{"type":"v1","addr":"172.21.15.170:6815","nonce":86580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86580},{"type":"v1","addr":"172.21.15.170:6813","nonce":86580}]},"public_addr":"172.21.15.170:6809/86580","cluster_addr":"172.21.15.170:6811/86580","heartbeat_back_addr":"172.21.15.170:6815/86580","heartbeat_front_addr":"172.21.15.170:6813/86580","state":["exists","up"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":86581},{"type":"v1","addr":"172.21.15.170:6817","nonce":86581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86581},{"type":"v1","addr":"172.21.15.170:6819","nonce":86581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86581},{"type":"v1","addr":"172.21.15.170:6823","nonce":86581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86581},{"type":"v1","addr":"172.21.15.170:6821","nonce":86581}]},"public_addr":"172.21.15.170:6817/86581","cluster_addr":"172.21.15.170:6819/86581","heartbeat_back_addr":"172.21.15.170:6823/86581","heartbeat_front_addr":"172.21.15.170:6821/86581","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.528486+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.550125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.433234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.495896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.515277+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.545278+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:01:01.990 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-12-01T22:00:51.401944+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-12-01T22:00:53.575133+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-12-01T22:01:01.990 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2023-12-01T22:01:02.296 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2023-12-01T22:01:02.310 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-12-01T22:01:02.616 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 8 2023-12-01T22:01:02.629 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2023-12-01T22:01:02.629 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:01:02.931 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:01:02.932 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:01:02.943 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-12-01T22:01:01.301187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":269082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.436530+0000","last_change":"2023-12-01T22:00:55.435954+0000","last_active":"2023-12-01T22:00:55.436530+0000","last_peered":"2023-12-01T22:00:55.436530+0000","last_clean":"2023-12-01T22:00:55.436530+0000","last_became_active":"2023-12-01T22:00:55.435598+0000","last_became_peered":"2023-12-01T22:00:55.435598+0000","last_unstale":"2023-12-01T22:00:55.436530+0000","last_undegraded":"2023-12-01T22:00:55.436530+0000","last_fullsized":"2023-12-01T22:00:55.436530+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383085+0000","last_change":"2023-12-01T22:00:55.380208+0000","last_active":"2023-12-01T22:00:55.383085+0000","last_peered":"2023-12-01T22:00:55.383085+0000","last_clean":"2023-12-01T22:00:55.383085+0000","last_became_active":"2023-12-01T22:00:55.379870+0000","last_became_peered":"2023-12-01T22:00:55.379870+0000","last_unstale":"2023-12-01T22:00:55.383085+0000","last_undegraded":"2023-12-01T22:00:55.383085+0000","last_fullsized":"2023-12-01T22:00:55.383085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412205+0000","last_change":"2023-12-01T22:00:55.411681+0000","last_active":"2023-12-01T22:00:55.412205+0000","last_peered":"2023-12-01T22:00:55.412205+0000","last_clean":"2023-12-01T22:00:55.412205+0000","last_became_active":"2023-12-01T22:00:55.411272+0000","last_became_peered":"2023-12-01T22:00:55.411272+0000","last_unstale":"2023-12-01T22:00:55.412205+0000","last_undegraded":"2023-12-01T22:00:55.412205+0000","last_fullsized":"2023-12-01T22:00:55.412205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383697+0000","last_change":"2023-12-01T22:00:55.383115+0000","last_active":"2023-12-01T22:00:55.383697+0000","last_peered":"2023-12-01T22:00:55.383697+0000","last_clean":"2023-12-01T22:00:55.383697+0000","last_became_active":"2023-12-01T22:00:55.382740+0000","last_became_peered":"2023-12-01T22:00:55.382740+0000","last_unstale":"2023-12-01T22:00:55.383697+0000","last_undegraded":"2023-12-01T22:00:55.383697+0000","last_fullsized":"2023-12-01T22:00:55.383697+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.373530+0000","last_change":"2023-12-01T22:00:53.374514+0000","last_active":"2023-12-01T22:00:55.373530+0000","last_peered":"2023-12-01T22:00:55.373530+0000","last_clean":"2023-12-01T22:00:55.373530+0000","last_became_active":"2023-12-01T22:00:53.374199+0000","last_became_peered":"2023-12-01T22:00:53.374199+0000","last_unstale":"2023-12-01T22:00:55.373530+0000","last_undegraded":"2023-12-01T22:00:55.373530+0000","last_fullsized":"2023-12-01T22:00:55.373530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.384148+0000","last_change":"2023-12-01T22:00:55.383420+0000","last_active":"2023-12-01T22:00:55.384148+0000","last_peered":"2023-12-01T22:00:55.384148+0000","last_clean":"2023-12-01T22:00:55.384148+0000","last_became_active":"2023-12-01T22:00:55.382971+0000","last_became_peered":"2023-12-01T22:00:55.382971+0000","last_unstale":"2023-12-01T22:00:55.384148+0000","last_undegraded":"2023-12-01T22:00:55.384148+0000","last_fullsized":"2023-12-01T22:00:55.384148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.386532+0000","last_change":"2023-12-01T22:00:55.383868+0000","last_active":"2023-12-01T22:00:55.386532+0000","last_peered":"2023-12-01T22:00:55.386532+0000","last_clean":"2023-12-01T22:00:55.386532+0000","last_became_active":"2023-12-01T22:00:55.383492+0000","last_became_peered":"2023-12-01T22:00:55.383492+0000","last_unstale":"2023-12-01T22:00:55.386532+0000","last_undegraded":"2023-12-01T22:00:55.386532+0000","last_fullsized":"2023-12-01T22:00:55.386532+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383918+0000","last_change":"2023-12-01T22:00:55.383150+0000","last_active":"2023-12-01T22:00:55.383918+0000","last_peered":"2023-12-01T22:00:55.383918+0000","last_clean":"2023-12-01T22:00:55.383918+0000","last_became_active":"2023-12-01T22:00:55.382812+0000","last_became_peered":"2023-12-01T22:00:55.382812+0000","last_unstale":"2023-12-01T22:00:55.383918+0000","last_undegraded":"2023-12-01T22:00:55.383918+0000","last_fullsized":"2023-12-01T22:00:55.383918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412954+0000","last_change":"2023-12-01T22:00:55.411524+0000","last_active":"2023-12-01T22:00:55.412954+0000","last_peered":"2023-12-01T22:00:55.412954+0000","last_clean":"2023-12-01T22:00:55.412954+0000","last_became_active":"2023-12-01T22:00:55.411153+0000","last_became_peered":"2023-12-01T22:00:55.411153+0000","last_unstale":"2023-12-01T22:00:55.412954+0000","last_undegraded":"2023-12-01T22:00:55.412954+0000","last_fullsized":"2023-12-01T22:00:55.412954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:01:02.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:01:03.249 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:01:03.251 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:01:03.262 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-12-01T22:01:01.301187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1783416,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564447624,"statfs":{"total":579820584960,"available":577994366976,"internally_reserved":0,"allocated":860160,"data_stored":269082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.436530+0000","last_change":"2023-12-01T22:00:55.435954+0000","last_active":"2023-12-01T22:00:55.436530+0000","last_peered":"2023-12-01T22:00:55.436530+0000","last_clean":"2023-12-01T22:00:55.436530+0000","last_became_active":"2023-12-01T22:00:55.435598+0000","last_became_peered":"2023-12-01T22:00:55.435598+0000","last_unstale":"2023-12-01T22:00:55.436530+0000","last_undegraded":"2023-12-01T22:00:55.436530+0000","last_fullsized":"2023-12-01T22:00:55.436530+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383085+0000","last_change":"2023-12-01T22:00:55.380208+0000","last_active":"2023-12-01T22:00:55.383085+0000","last_peered":"2023-12-01T22:00:55.383085+0000","last_clean":"2023-12-01T22:00:55.383085+0000","last_became_active":"2023-12-01T22:00:55.379870+0000","last_became_peered":"2023-12-01T22:00:55.379870+0000","last_unstale":"2023-12-01T22:00:55.383085+0000","last_undegraded":"2023-12-01T22:00:55.383085+0000","last_fullsized":"2023-12-01T22:00:55.383085+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412205+0000","last_change":"2023-12-01T22:00:55.411681+0000","last_active":"2023-12-01T22:00:55.412205+0000","last_peered":"2023-12-01T22:00:55.412205+0000","last_clean":"2023-12-01T22:00:55.412205+0000","last_became_active":"2023-12-01T22:00:55.411272+0000","last_became_peered":"2023-12-01T22:00:55.411272+0000","last_unstale":"2023-12-01T22:00:55.412205+0000","last_undegraded":"2023-12-01T22:00:55.412205+0000","last_fullsized":"2023-12-01T22:00:55.412205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383697+0000","last_change":"2023-12-01T22:00:55.383115+0000","last_active":"2023-12-01T22:00:55.383697+0000","last_peered":"2023-12-01T22:00:55.383697+0000","last_clean":"2023-12-01T22:00:55.383697+0000","last_became_active":"2023-12-01T22:00:55.382740+0000","last_became_peered":"2023-12-01T22:00:55.382740+0000","last_unstale":"2023-12-01T22:00:55.383697+0000","last_undegraded":"2023-12-01T22:00:55.383697+0000","last_fullsized":"2023-12-01T22:00:55.383697+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.373530+0000","last_change":"2023-12-01T22:00:53.374514+0000","last_active":"2023-12-01T22:00:55.373530+0000","last_peered":"2023-12-01T22:00:55.373530+0000","last_clean":"2023-12-01T22:00:55.373530+0000","last_became_active":"2023-12-01T22:00:53.374199+0000","last_became_peered":"2023-12-01T22:00:53.374199+0000","last_unstale":"2023-12-01T22:00:55.373530+0000","last_undegraded":"2023-12-01T22:00:55.373530+0000","last_fullsized":"2023-12-01T22:00:55.373530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.384148+0000","last_change":"2023-12-01T22:00:55.383420+0000","last_active":"2023-12-01T22:00:55.384148+0000","last_peered":"2023-12-01T22:00:55.384148+0000","last_clean":"2023-12-01T22:00:55.384148+0000","last_became_active":"2023-12-01T22:00:55.382971+0000","last_became_peered":"2023-12-01T22:00:55.382971+0000","last_unstale":"2023-12-01T22:00:55.384148+0000","last_undegraded":"2023-12-01T22:00:55.384148+0000","last_fullsized":"2023-12-01T22:00:55.384148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.386532+0000","last_change":"2023-12-01T22:00:55.383868+0000","last_active":"2023-12-01T22:00:55.386532+0000","last_peered":"2023-12-01T22:00:55.386532+0000","last_clean":"2023-12-01T22:00:55.386532+0000","last_became_active":"2023-12-01T22:00:55.383492+0000","last_became_peered":"2023-12-01T22:00:55.383492+0000","last_unstale":"2023-12-01T22:00:55.386532+0000","last_undegraded":"2023-12-01T22:00:55.386532+0000","last_fullsized":"2023-12-01T22:00:55.386532+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.383918+0000","last_change":"2023-12-01T22:00:55.383150+0000","last_active":"2023-12-01T22:00:55.383918+0000","last_peered":"2023-12-01T22:00:55.383918+0000","last_clean":"2023-12-01T22:00:55.383918+0000","last_became_active":"2023-12-01T22:00:55.382812+0000","last_became_peered":"2023-12-01T22:00:55.382812+0000","last_unstale":"2023-12-01T22:00:55.383918+0000","last_undegraded":"2023-12-01T22:00:55.383918+0000","last_fullsized":"2023-12-01T22:00:55.383918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-01T22:00:55.412954+0000","last_change":"2023-12-01T22:00:55.411524+0000","last_active":"2023-12-01T22:00:55.412954+0000","last_peered":"2023-12-01T22:00:55.412954+0000","last_clean":"2023-12-01T22:00:55.412954+0000","last_became_active":"2023-12-01T22:00:55.411153+0000","last_became_peered":"2023-12-01T22:00:55.411153+0000","last_unstale":"2023-12-01T22:00:55.412954+0000","last_undegraded":"2023-12-01T22:00:55.412954+0000","last_fullsized":"2023-12-01T22:00:55.412954+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297232,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074608,"statfs":{"total":96636764160,"available":96332398592,"internally_reserved":0,"allocated":139264,"data_stored":43202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297244,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074596,"statfs":{"total":96636764160,"available":96332386304,"internally_reserved":0,"allocated":151552,"data_stored":48137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:01:03.263 INFO:tasks.mon_thrash.ceph_manager:clean! 2023-12-01T22:01:03.264 DEBUG:tasks.mon_thrash:Yielding 2023-12-01T22:01:03.264 INFO:teuthology.run_tasks:Running task workunit... 2023-12-01T22:01:03.276 INFO:tasks.workunit:Pulling workunits from ref 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:01:03.277 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-01T22:01:03.277 DEBUG:teuthology.orchestra.run.smithi170:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-01T22:01:03.277 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2023-12-01T22:01:03.278 INFO:tasks.mon_thrash.mon_thrasher:seed: 1701468063, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2023-12-01T22:01:03.278 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:01:03.278 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:01:03.297 INFO:teuthology.orchestra.run.smithi170.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-01T22:01:03.297 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-01T22:01:03.354 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-01T22:01:03.354 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-01T22:01:03.411 INFO:tasks.workunit:timeout=3h 2023-12-01T22:01:03.412 INFO:tasks.workunit:cleanup=True 2023-12-01T22:01:03.412 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-12-01T22:01:03.469 INFO:tasks.workunit.client.0.smithi170.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-01T22:01:03.596 INFO:teuthology.orchestra.run.smithi039.stderr:2023-12-01T22:01:03.595+0000 7f83ba529700 0 --2- 172.21.15.39:0/995394070 >> v2:172.21.15.39:3300/0 conn(0x7f83b4129fc0 0x7f83b4133c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-01T22:01:03.853 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:03.866 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:01:03.866 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:01:03.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:01:03.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:01:04.005 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 8, 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 22, 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:04.006 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.007 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:04.008 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.009 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:04.010 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.011 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.013 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:04.016 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.017 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.017 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:04.017 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:04.018 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:04.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:01:04.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 8, 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:04.158 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 22, 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:04.159 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:04.160 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.161 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:04.162 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:04.163 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:04.164 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.165 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.166 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.167 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.168 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:04.169 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:04.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:04.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:01:04.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:01:04.314 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:04.314 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:04.314 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:04.314 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 8, 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 22, 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.315 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.316 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:04.317 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:04.318 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.320 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:04.321 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.322 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:04.323 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.324 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:04.325 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:04.327 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:04.328 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-12-01T22:01:04.328 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-01T22:01:04.328 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-12-01T22:01:04.328 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-12-01T22:01:04.328 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-01T22:01:04.328 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:01:04.338 INFO:tasks.ceph.mon.a:Stopped 2023-12-01T22:01:04.338 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:01:04.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:04.748 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:07.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:09.603 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:12.623 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:13.010 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:16.025 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:19.423 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:19.437 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:01:19.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-01T22:01:19.438 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 10, 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:19.581 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:19.582 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:19.583 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:19.585 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:19.586 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:19.587 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:19.588 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:19.589 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:19.590 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:19.591 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:19.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.591 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:19.591 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:19.591 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.592 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:19.593 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.595 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:19.596 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.597 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:19.598 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:19.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.598 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.598 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.598 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:19.598 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.599 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:19.600 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:19.602 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:01:19.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-12-01T22:01:19.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:01:19.743 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 10, 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:19.744 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:19.745 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:19.746 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:19.747 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:19.748 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:19.749 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:19.750 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:19.751 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.754 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:19.755 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:19.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:19.756 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:19.756 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:19.757 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:01:19.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:01:39.758 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-12-01T22:01:39.758 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-12-01T22:01:39.758 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-01T22:01:39.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-01T22:01:39.762 INFO:tasks.ceph.mon.a:Started 2023-12-01T22:01:39.762 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:01:39.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:49.927 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:52.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:53.333 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:53.346 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:01:53.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:01:53.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:01:53.449 INFO:tasks.workunit.client.0.smithi170.stderr:Note: switching to '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'. 2023-12-01T22:01:53.449 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.449 INFO:tasks.workunit.client.0.smithi170.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:state without impacting any branches by switching back to a branch. 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: git switch -c 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:Or undo this operation with: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: git switch - 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr: 2023-12-01T22:01:53.450 INFO:tasks.workunit.client.0.smithi170.stderr:HEAD is now at 0dc074ce0b0 Merge branch 'wip-63588-pacific' of https://github.com/mchangir/ceph into wip-yuri8-testing-2023-11-29-0706-pacific 2023-12-01T22:01:53.455 DEBUG:teuthology.orchestra.run.smithi170:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 22, 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:53.497 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:53.498 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:53.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:01:53.499 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:01:53.521 INFO:tasks.workunit.client.0.smithi170.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-01T22:01:53.523 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-01T22:01:53.523 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 22, 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:53.640 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.641 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:53.642 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:53.643 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:53.644 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.646 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.647 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.648 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.649 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:53.651 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:53.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:53.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:01:53.652 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:01:53.792 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:53.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 22, 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.793 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.794 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:53.795 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:53.796 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:53.797 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.800 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.802 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:53.803 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:53.804 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:53.805 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:53.806 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:01:53.806 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:01:54.075 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-01T22:01:54.139 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-01T22:01:54.175 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-01T22:01:54.176 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-01T22:01:54.177 INFO:tasks.workunit.client.0.smithi170.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-01T22:01:54.179 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.178+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:01:54.179 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.178+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=1,health=12,kv=7,logm=66,mdsmap=3,mgr=4} crc {auth=1529440529,config=1228700967,health=2406479829,kv=2377187526,logm=3185026592,mdsmap=1531151339,mgr=3348470407}) 2023-12-01T22:01:54.180 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.178+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=7,logm=64,mdsmap=3,mgr=4} crc {auth=3101943858,config=3140514009,health=2406479829,kv=2377187526,logm=1504425925,mdsmap=1531151339,mgr=305193480}) 2023-12-01T22:01:54.180 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.178+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:01:54.180 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.178+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=1,health=12,kv=7,logm=66,mdsmap=3,mgr=4} crc {auth=1529440529,config=1228700967,health=2406479829,kv=2377187526,logm=3185026592,mdsmap=1531151339,mgr=3348470407}) 2023-12-01T22:01:54.180 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.178+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=11,kv=7,logm=66,mdsmap=2,mgr=4} crc {auth=3101943858,config=3140514009,health=420257518,kv=2377187526,logm=486405977,mdsmap=3585761405,mgr=305193480}) 2023-12-01T22:01:54.181 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.180+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:01:54.181 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.180+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=34} crc {mgr=3541143443,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3893032179,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=577945228,osdmap=767024148}) 2023-12-01T22:01:54.181 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.180+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=33} crc {mgr=3541143443,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3893032179,mon_config_key=161422294,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=577945228,osdmap=196398737}) 2023-12-01T22:01:54.182 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.180+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:01:54.182 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.180+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=34} crc {mgr=3541143443,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3893032179,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=577945228,osdmap=767024148}) 2023-12-01T22:01:54.182 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:01:54.180+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=32} crc {mgr=3541143443,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1166359038,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=577945228,osdmap=2572824808}) 2023-12-01T22:01:54.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:01:54.212 INFO:tasks.workunit.client.0.smithi170.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-01T22:01:54.215 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-01T22:01:54.215 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-01T22:01:54.271 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2023-12-01T22:01:54.272 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2023-12-01T22:01:54.273 DEBUG:teuthology.orchestra.run.smithi170:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2023-12-01T22:01:54.329 INFO:tasks.workunit.client.0.smithi170.stderr:+ TEST_POOL=testpool1234 2023-12-01T22:01:54.329 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2023-12-01T22:01:55.194 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:01:55.195 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:55.574 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:55.588 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:01:55.588 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:01:55.588 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:01:55.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:01:55.661 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'testpool1234' already exists 2023-12-01T22:01:55.670 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2023-12-01T22:01:55.727 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 22, 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:55.728 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:55.729 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:55.730 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:55.731 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.732 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:55.734 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:55.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:55.737 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:55.738 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:55.739 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:55.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:01:55.740 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 22, 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.881 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:55.892 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:55.892 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:55.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:01:55.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:01:56.033 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 22, 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:01:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:01:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:01:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:01:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:01:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:01:56.045 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:01:56.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:01:56.047 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-12-01T22:01:56.047 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-01T22:01:56.047 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-12-01T22:01:56.047 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-12-01T22:01:56.047 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-01T22:01:56.047 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:01:56.057 INFO:tasks.ceph.mon.c:Stopped 2023-12-01T22:01:56.058 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:01:56.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:56.470 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:01:59.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:01:59.870 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:02.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:02:04.105 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:07.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:02:11.187 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:11.201 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:02:11.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-01T22:02:11.202 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:02:11.347 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 26, 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: 1 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:11.348 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:11.349 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:11.350 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:11.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.353 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:11.354 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.355 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:11.356 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:11.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:11.358 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:11.359 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:02:11.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-12-01T22:02:11.359 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 26, 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:11.502 INFO:teuthology.orchestra.run.smithi039.stdout: 1 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:11.503 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:11.504 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:11.505 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.506 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:11.508 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:11.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:11.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 6 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.512 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:11.513 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:11.514 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:11.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:02:11.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:02:12.921 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 3 pg_num_min to 2 2023-12-01T22:02:12.932 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool get testpool1234 pg_num_min 2023-12-01T22:02:12.932 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 2 2023-12-01T22:02:13.322 INFO:tasks.workunit.client.0.smithi170.stdout:pg_num_min: 2 2023-12-01T22:02:13.323 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2023-12-01T22:02:14.930 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 3 pg_num_max to 33 2023-12-01T22:02:14.940 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool get testpool1234 pg_num_max 2023-12-01T22:02:14.940 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 33 2023-12-01T22:02:15.947 INFO:tasks.workunit.client.0.smithi170.stdout:pg_num_max: 33 2023-12-01T22:02:15.948 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2023-12-01T22:02:15.948 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:15.948 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2023-12-01T22:02:17.159 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2023-12-01T22:02:17.161 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:17.161 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2023-12-01T22:02:17.162 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:17.162 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2023-12-01T22:02:17.437 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2023-12-01T22:02:17.439 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:17.439 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2023-12-01T22:02:17.439 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:17.439 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num 1 2023-12-01T22:02:17.715 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2023-12-01T22:02:17.717 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:17.717 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2023-12-01T22:02:17.717 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:17.718 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num 44 2023-12-01T22:02:18.609 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2023-12-01T22:02:18.611 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:18.611 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2023-12-01T22:02:19.965 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 3 pg_num_min to 0 2023-12-01T22:02:19.977 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2023-12-01T22:02:19.977 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:19.977 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool get testpool1234 pg_num_min 2023-12-01T22:02:20.252 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2023-12-01T22:02:20.253 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:20.254 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2023-12-01T22:02:21.968 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 3 pg_num_max to 0 2023-12-01T22:02:21.978 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2023-12-01T22:02:21.978 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:21.978 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool get testpool1234 pg_num_max 2023-12-01T22:02:22.254 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2023-12-01T22:02:22.255 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:22.255 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2023-12-01T22:02:23.083 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'testpool1234' does not exist 2023-12-01T22:02:23.093 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-ruleset 2023-12-01T22:02:23.093 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:23.093 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-ruleset 2023-12-01T22:02:23.370 INFO:tasks.workunit.client.0.smithi170.stderr:foo-ruleset not valid: foo-ruleset doesn't represent a float 2023-12-01T22:02:23.370 INFO:tasks.workunit.client.0.smithi170.stderr:Invalid command: unused arguments: ['foo-ruleset'] 2023-12-01T22:02:23.370 INFO:tasks.workunit.client.0.smithi170.stderr:osd pool create [] [] [replicated|erasure] [] [] [] [] [] [] [on|off|warn] [--bulk] [] [] : create pool 2023-12-01T22:02:23.370 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: invalid command 2023-12-01T22:02:23.371 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:23.371 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create foo 123 123 replicated 2023-12-01T22:02:24.098 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'foo' already exists 2023-12-01T22:02:24.107 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create fooo 123 123 erasure default 2023-12-01T22:02:27.124 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'fooo' already exists 2023-12-01T22:02:27.133 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create foooo 123 2023-12-01T22:02:29.131 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'foooo' already exists 2023-12-01T22:02:29.141 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create foo 123 2023-12-01T22:02:29.522 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'foo' already exists 2023-12-01T22:02:29.532 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2023-12-01T22:02:31.034 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 4 size to 1 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_config_value foo min_size 1 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:+ local pool_name config_opt expected_val val 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:+ pool_name=foo 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:+ config_opt=min_size 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:+ expected_val=1 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:++ get_config_value_or_die foo min_size 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:++ local pool_name config_opt raw val 2023-12-01T22:02:31.053 INFO:tasks.workunit.client.0.smithi170.stderr:++ pool_name=foo 2023-12-01T22:02:31.054 INFO:tasks.workunit.client.0.smithi170.stderr:++ config_opt=min_size 2023-12-01T22:02:31.054 INFO:tasks.workunit.client.0.smithi170.stderr:+++ ceph osd pool get foo min_size 2023-12-01T22:02:31.448 INFO:tasks.workunit.client.0.smithi170.stderr:++ raw='min_size: 1' 2023-12-01T22:02:31.448 INFO:tasks.workunit.client.0.smithi170.stderr:++ [[ 0 -ne 0 ]] 2023-12-01T22:02:31.448 INFO:tasks.workunit.client.0.smithi170.stderr:+++ echo min_size: 1 2023-12-01T22:02:31.448 INFO:tasks.workunit.client.0.smithi170.stderr:+++ sed -e 's/[{} "]//g' 2023-12-01T22:02:31.449 INFO:tasks.workunit.client.0.smithi170.stderr:++ raw=min_size:1 2023-12-01T22:02:31.449 INFO:tasks.workunit.client.0.smithi170.stderr:+++ echo min_size:1 2023-12-01T22:02:31.450 INFO:tasks.workunit.client.0.smithi170.stderr:+++ cut -f2 -d: 2023-12-01T22:02:31.450 INFO:tasks.workunit.client.0.smithi170.stderr:++ val=1 2023-12-01T22:02:31.451 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo 1 2023-12-01T22:02:31.451 INFO:tasks.workunit.client.0.smithi170.stderr:++ return 0 2023-12-01T22:02:31.451 INFO:tasks.workunit.client.0.smithi170.stderr:+ val=1 2023-12-01T22:02:31.451 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ 1 != \1 ]] 2023-12-01T22:02:31.451 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 4 2023-12-01T22:02:31.515 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-12-01T22:02:31.515 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-12-01T22:02:31.515 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-01T22:02:31.516 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-01T22:02:31.518 INFO:tasks.ceph.mon.c:Started 2023-12-01T22:02:31.518 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:02:31.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:02:31.903 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:34.070 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 4 size to 4 2023-12-01T22:02:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_config_value foo min_size 2 2023-12-01T22:02:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ local pool_name config_opt expected_val val 2023-12-01T22:02:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ pool_name=foo 2023-12-01T22:02:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ config_opt=min_size 2023-12-01T22:02:34.086 INFO:tasks.workunit.client.0.smithi170.stderr:+ expected_val=2 2023-12-01T22:02:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:++ get_config_value_or_die foo min_size 2023-12-01T22:02:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:++ local pool_name config_opt raw val 2023-12-01T22:02:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:++ pool_name=foo 2023-12-01T22:02:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:++ config_opt=min_size 2023-12-01T22:02:34.087 INFO:tasks.workunit.client.0.smithi170.stderr:+++ ceph osd pool get foo min_size 2023-12-01T22:02:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:++ raw='min_size: 2' 2023-12-01T22:02:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:++ [[ 0 -ne 0 ]] 2023-12-01T22:02:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:+++ echo min_size: 2 2023-12-01T22:02:34.500 INFO:tasks.workunit.client.0.smithi170.stderr:+++ sed -e 's/[{} "]//g' 2023-12-01T22:02:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:++ raw=min_size:2 2023-12-01T22:02:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:+++ echo min_size:2 2023-12-01T22:02:34.501 INFO:tasks.workunit.client.0.smithi170.stderr:+++ cut -f2 -d: 2023-12-01T22:02:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ val=2 2023-12-01T22:02:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo 2 2023-12-01T22:02:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:++ return 0 2023-12-01T22:02:34.502 INFO:tasks.workunit.client.0.smithi170.stderr:+ val=2 2023-12-01T22:02:34.503 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ 2 != \2 ]] 2023-12-01T22:02:34.503 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 10 2023-12-01T22:02:34.917 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:02:40.001 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:40.929 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 4 size to 10 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_config_value foo min_size 5 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:+ local pool_name config_opt expected_val val 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:+ pool_name=foo 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:+ config_opt=min_size 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:+ expected_val=5 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ get_config_value_or_die foo min_size 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ local pool_name config_opt raw val 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ pool_name=foo 2023-12-01T22:02:40.945 INFO:tasks.workunit.client.0.smithi170.stderr:++ config_opt=min_size 2023-12-01T22:02:40.946 INFO:tasks.workunit.client.0.smithi170.stderr:+++ ceph osd pool get foo min_size 2023-12-01T22:02:41.367 INFO:tasks.workunit.client.0.smithi170.stderr:++ raw='min_size: 5' 2023-12-01T22:02:41.367 INFO:tasks.workunit.client.0.smithi170.stderr:++ [[ 0 -ne 0 ]] 2023-12-01T22:02:41.367 INFO:tasks.workunit.client.0.smithi170.stderr:+++ echo min_size: 5 2023-12-01T22:02:41.367 INFO:tasks.workunit.client.0.smithi170.stderr:+++ sed -e 's/[{} "]//g' 2023-12-01T22:02:41.368 INFO:tasks.workunit.client.0.smithi170.stderr:++ raw=min_size:5 2023-12-01T22:02:41.368 INFO:tasks.workunit.client.0.smithi170.stderr:+++ echo min_size:5 2023-12-01T22:02:41.369 INFO:tasks.workunit.client.0.smithi170.stderr:+++ cut -f2 -d: 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:++ val=5 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:++ echo 5 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:++ return 0 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:+ val=5 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:+ [[ 5 != \5 ]] 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set foo size 0 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:41.370 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 0 2023-12-01T22:02:41.646 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: pool size must be between 1 and 10 2023-12-01T22:02:41.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:41.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set foo size 20 2023-12-01T22:02:41.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:41.648 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 20 2023-12-01T22:02:41.927 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: pool size must be between 1 and 10 2023-12-01T22:02:41.927 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:02:41.927 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 3 2023-12-01T22:02:43.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:02:43.444 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:43.458 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:02:43.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:02:43.458 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 32, 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:43.598 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 2, 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.599 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.600 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:43.601 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.602 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:43.603 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.605 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.606 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.607 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.608 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:43.609 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:43.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:02:43.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:02:43.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:02:43.750 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:43.750 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 32, 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.751 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.752 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:43.753 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.754 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.756 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:43.758 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.759 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:43.760 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.761 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.761 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:43.761 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:43.762 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:02:43.762 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:02:43.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:02:43.902 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:43.902 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:43.902 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:43.902 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:02:43.902 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 32, 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.903 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.904 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:43.905 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:43.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.907 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.908 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:43.910 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.911 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.912 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:43.913 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:43.914 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:43.914 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:43.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:43.914 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:43.914 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:43.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:02:43.915 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:02:43.916 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:02:44.033 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 4 size to 3 2023-12-01T22:02:44.051 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd getcrushmap -o crush 2023-12-01T22:02:44.351 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:02:44.352 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=11,kv=7,logm=71} crc {auth=623469491,config=3140514009,health=3172947059,kv=2377187526,logm=3684159075}) 2023-12-01T22:02:44.352 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=7,logm=71} crc {auth=2051422290,config=3140514009,health=2335858825,kv=2377187526,logm=560232207}) 2023-12-01T22:02:44.352 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:02:44.352 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=11,kv=7,logm=71} crc {auth=623469491,config=3140514009,health=3172947059,kv=2377187526,logm=3684159075}) 2023-12-01T22:02:44.352 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=12,kv=7,logm=71} crc {auth=2183066464,config=3140514009,health=1148745368,kv=2377187526,logm=1716030132}) 2023-12-01T22:02:44.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:02:44.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=53,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=9} crc {logm=1020127342,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=4221979585,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=1370131380,osdmap=1706943030}) 2023-12-01T22:02:44.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=55,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=9} crc {logm=1003460141,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3630820817,mon_config_key=2250838263,monmap=3140514009,osd_metadata=2309343068,osd_pg_creating=1370131380,osdmap=91560847}) 2023-12-01T22:02:44.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:02:44.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=53,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=9} crc {logm=1020127342,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=4221979585,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=1370131380,osdmap=1706943030}) 2023-12-01T22:02:44.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=10} crc {logm=4088276258,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=2167088779,mon_config_key=2250838263,monmap=342008835,osd_metadata=2309343068,osd_pg_creating=1370131380,osdmap=3862881164}) 2023-12-01T22:02:44.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.354+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:02:44.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.354+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=3739688066}) 2023-12-01T22:02:44.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.354+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=68} crc {osdmap=709358995}) 2023-12-01T22:02:44.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.354+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:02:44.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.354+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=3739688066}) 2023-12-01T22:02:44.356 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:02:44.354+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=1065917554}) 2023-12-01T22:02:44.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:02:44.463 INFO:tasks.workunit.client.0.smithi170.stderr:5 2023-12-01T22:02:44.473 INFO:tasks.workunit.client.0.smithi170.stderr:+ crushtool -d crush -o crush.txt 2023-12-01T22:02:44.485 INFO:tasks.workunit.client.0.smithi170.stderr:+ sed -i 's/max_size 10/max_size 3/' crush.txt 2023-12-01T22:02:44.486 INFO:tasks.workunit.client.0.smithi170.stderr:+ crushtool -c crush.txt -o crush.new 2023-12-01T22:02:44.498 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd setcrushmap -i crush.new 2023-12-01T22:02:45.366 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:02:45.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:02:45.739 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:02:45.752 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:02:45.752 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:02:45.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:02:45.753 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:02:45.892 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:45.892 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:45.892 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:45.892 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:02:45.892 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 32, 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 5, 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:45.893 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:45.894 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:45.895 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:45.896 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:45.898 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:45.899 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:45.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:02:45.902 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:45.903 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:45.904 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:02:45.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:02:45.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:02:46.035 INFO:tasks.workunit.client.0.smithi170.stderr:7 2023-12-01T22:02:46.046 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool set foo size 4 2023-12-01T22:02:46.046 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:02:46.046 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set foo size 4 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 32, 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.048 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 5, 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:46.049 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:46.050 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:46.051 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:46.052 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.053 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:46.054 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.056 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:46.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:46.058 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:46.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:02:46.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:02:46.062 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:02:46.203 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:02:46.203 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:46.203 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:46.203 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 32, 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 5, 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:46.204 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:46.205 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:02:46.206 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:02:46.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:02:46.208 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:46.209 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:02:46.210 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:02:46.211 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:02:46.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:46.213 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:46.214 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:02:46.214 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.214 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:02:46.214 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:02:46.217 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:02:46.218 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:02:46.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-12-01T22:02:46.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2023-12-01T22:02:46.219 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-12-01T22:02:46.219 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-12-01T22:02:46.219 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-01T22:02:46.219 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:02:46.228 INFO:tasks.ceph.mon.c:Stopped 2023-12-01T22:02:46.228 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-12-01T22:02:46.228 INFO:tasks.ceph.mon.a:Sent signal 19 2023-12-01T22:02:46.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-12-01T22:03:01.229 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-12-01T22:03:01.229 INFO:tasks.ceph.mon.a:Sent signal 18 2023-12-01T22:03:01.229 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:03:01.230 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:03:06.286 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: pool size is bigger than the crush rule max size 2023-12-01T22:03:06.287 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:06.288 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd setcrushmap -i crush 2023-12-01T22:03:06.542 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:03:06.556 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:03:06.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-01T22:03:06.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:03:06.697 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:06.697 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:06.697 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 36, 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: 1 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:06.698 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:06.699 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:06.700 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.701 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:06.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:06.704 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:06.705 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.706 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.707 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:03:06.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.708 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.708 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:06.708 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:06.709 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:03:06.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-12-01T22:03:06.709 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:03:06.849 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:06.849 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 36, 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: 1 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:06.850 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:06.851 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:06.852 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.853 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:06.854 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:06.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:06.856 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:06.857 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.858 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.859 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:06.860 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:06.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:03:06.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:03:08.279 INFO:tasks.workunit.client.0.smithi170.stderr:9 2023-12-01T22:03:08.289 INFO:tasks.workunit.client.0.smithi170.stderr:+ rm -f crush crush.txt crush.new 2023-12-01T22:03:08.290 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool delete foo 2023-12-01T22:03:08.290 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:08.290 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo 2023-12-01T22:03:08.572 INFO:tasks.workunit.client.0.smithi170.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2023-12-01T22:03:08.574 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:08.574 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool delete foo foo 2023-12-01T22:03:08.574 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:08.574 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo foo 2023-12-01T22:03:08.848 INFO:tasks.workunit.client.0.smithi170.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2023-12-01T22:03:08.850 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:08.850 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool delete foo foo --force 2023-12-01T22:03:08.850 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:08.850 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo foo --force 2023-12-01T22:03:09.120 INFO:tasks.workunit.client.0.smithi170.stderr:--force not valid: --force not one of 'true', 'false' 2023-12-01T22:03:09.121 INFO:tasks.workunit.client.0.smithi170.stderr:Invalid command: unused arguments: ['--force'] 2023-12-01T22:03:09.121 INFO:tasks.workunit.client.0.smithi170.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-12-01T22:03:09.121 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: invalid command 2023-12-01T22:03:09.122 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:09.122 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2023-12-01T22:03:09.122 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:09.122 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2023-12-01T22:03:10.012 INFO:tasks.workunit.client.0.smithi170.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2023-12-01T22:03:10.012 INFO:tasks.workunit.client.0.smithi170.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2023-12-01T22:03:10.012 INFO:tasks.workunit.client.0.smithi170.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-12-01T22:03:10.012 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: invalid command 2023-12-01T22:03:10.014 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:10.014 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2023-12-01T22:03:10.014 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:10.014 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2023-12-01T22:03:10.296 INFO:tasks.workunit.client.0.smithi170.stderr:foo not valid: foo not one of 'true', 'false' 2023-12-01T22:03:10.296 INFO:tasks.workunit.client.0.smithi170.stderr:Invalid command: unused arguments: ['foo'] 2023-12-01T22:03:10.296 INFO:tasks.workunit.client.0.smithi170.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2023-12-01T22:03:10.296 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: invalid command 2023-12-01T22:03:10.298 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:10.298 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2023-12-01T22:03:11.363 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'foooo' does not exist 2023-12-01T22:03:11.372 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-12-01T22:03:13.232 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'fooo' does not exist 2023-12-01T22:03:13.243 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-12-01T22:03:15.260 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'foo' does not exist 2023-12-01T22:03:15.269 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2023-12-01T22:03:15.653 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'foo' does not exist 2023-12-01T22:03:15.662 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-12-01T22:03:16.045 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'fooo' does not exist 2023-12-01T22:03:16.055 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2023-12-01T22:03:17.058 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'fooo' does not exist 2023-12-01T22:03:17.067 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2023-12-01T22:03:17.456 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'fuggg' does not exist 2023-12-01T22:03:17.465 INFO:tasks.workunit.client.0.smithi170.stderr:+ echo OK 2023-12-01T22:03:17.466 INFO:tasks.workunit.client.0.smithi170.stdout:OK 2023-12-01T22:03:17.466 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-01T22:03:17.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-01T22:03:17.544 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2023-12-01T22:03:17.545 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2023-12-01T22:03:17.546 DEBUG:teuthology.orchestra.run.smithi170:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2023-12-01T22:03:17.604 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush dump 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout:{ 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "devices": [ 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 0, 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd.0", 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "class": "ssd" 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 1, 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd.1", 2023-12-01T22:03:17.986 INFO:tasks.workunit.client.0.smithi170.stdout: "class": "ssd" 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 2, 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd.2", 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "class": "ssd" 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 3, 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd.3", 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "class": "ssd" 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 4, 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd.4", 2023-12-01T22:03:17.987 INFO:tasks.workunit.client.0.smithi170.stdout: "class": "ssd" 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 5, 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd.5", 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "class": "ssd" 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: ], 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "types": [ 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 0, 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "osd" 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 1, 2023-12-01T22:03:17.988 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "host" 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 2, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "chassis" 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 3, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "rack" 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 4, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "row" 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 5, 2023-12-01T22:03:17.989 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "pdu" 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 6, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "pod" 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 7, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "room" 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 8, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "datacenter" 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.990 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 9, 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "zone" 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 10, 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "region" 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 11, 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "root" 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: ], 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "buckets": [ 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -1, 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "default", 2023-12-01T22:03:17.991 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 11, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "type_name": "root", 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 34560, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "alg": "straw2", 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "hash": "rjenkins1", 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "items": [ 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -3, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 0 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -5, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 1 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.992 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -2, 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "default~ssd", 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 11, 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "type_name": "root", 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 34560, 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "alg": "straw2", 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "hash": "rjenkins1", 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "items": [ 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -4, 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 0 2023-12-01T22:03:17.993 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -6, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 1 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -3, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "smithi039", 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 1, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "type_name": "host", 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "alg": "straw2", 2023-12-01T22:03:17.994 INFO:tasks.workunit.client.0.smithi170.stdout: "hash": "rjenkins1", 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "items": [ 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 1, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 0 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 2, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 1 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 0, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.995 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 2 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -4, 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "smithi039~ssd", 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 1, 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "type_name": "host", 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "alg": "straw2", 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "hash": "rjenkins1", 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "items": [ 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 1, 2023-12-01T22:03:17.996 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 0 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 2, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 1 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 0, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 2 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.997 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -5, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "smithi170", 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 1, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "type_name": "host", 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "alg": "straw2", 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "hash": "rjenkins1", 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "items": [ 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 3, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 0 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 4, 2023-12-01T22:03:17.998 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 1 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 5, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 2 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "id": -6, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "name": "smithi170~ssd", 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "type_id": 1, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "type_name": "host", 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 17280, 2023-12-01T22:03:17.999 INFO:tasks.workunit.client.0.smithi170.stdout: "alg": "straw2", 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "hash": "rjenkins1", 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "items": [ 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 3, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 0 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 4, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 1 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "id": 5, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "weight": 5760, 2023-12-01T22:03:18.000 INFO:tasks.workunit.client.0.smithi170.stdout: "pos": 2 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: ], 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "rules": [ 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_id": 0, 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_name": "replicated_rule", 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "ruleset": 0, 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "type": 1, 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "min_size": 1, 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "max_size": 10, 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "steps": [ 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "take", 2023-12-01T22:03:18.001 INFO:tasks.workunit.client.0.smithi170.stdout: "item": -1, 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "item_name": "default" 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "choose_firstn", 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 0, 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "type": "osd" 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "emit" 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_id": 1, 2023-12-01T22:03:18.002 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_name": "erasure-code", 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "ruleset": 1, 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "type": 3, 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "min_size": 3, 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "max_size": 3, 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "steps": [ 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "set_chooseleaf_tries", 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 5 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "set_choose_tries", 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 100 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "take", 2023-12-01T22:03:18.003 INFO:tasks.workunit.client.0.smithi170.stdout: "item": -1, 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "item_name": "default" 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "choose_indep", 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 0, 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "type": "osd" 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "emit" 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: ], 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "tunables": { 2023-12-01T22:03:18.004 INFO:tasks.workunit.client.0.smithi170.stdout: "choose_local_tries": 0, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "choose_local_fallback_tries": 0, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "choose_total_tries": 50, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "chooseleaf_descend_once": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "chooseleaf_vary_r": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "chooseleaf_stable": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "straw_calc_version": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "allowed_bucket_algs": 54, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "profile": "jewel", 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "optimal_tunables": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "legacy_tunables": 0, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "minimum_required_version": "jewel", 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "require_feature_tunables": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "require_feature_tunables2": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "has_v2_rules": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "require_feature_tunables3": 1, 2023-12-01T22:03:18.005 INFO:tasks.workunit.client.0.smithi170.stdout: "has_v3_rules": 0, 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout: "has_v4_buckets": 1, 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout: "require_feature_tunables5": 1, 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout: "has_v5_rules": 0 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout: "choose_args": {} 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout:} 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stdout: 2023-12-01T22:03:18.006 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule dump 2023-12-01T22:03:18.382 INFO:tasks.workunit.client.0.smithi170.stdout:[ 2023-12-01T22:03:18.382 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_id": 0, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_name": "replicated_rule", 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "ruleset": 0, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "type": 1, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "min_size": 1, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "max_size": 10, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "steps": [ 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "take", 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "item": -1, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "item_name": "default" 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "choose_firstn", 2023-12-01T22:03:18.383 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 0, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "type": "osd" 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "emit" 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_id": 1, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "rule_name": "erasure-code", 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "ruleset": 1, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "type": 3, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "min_size": 3, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "max_size": 3, 2023-12-01T22:03:18.384 INFO:tasks.workunit.client.0.smithi170.stdout: "steps": [ 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "set_chooseleaf_tries", 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 5 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "set_choose_tries", 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 100 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "take", 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "item": -1, 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "item_name": "default" 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.385 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "choose_indep", 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: "num": 0, 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: "type": "osd" 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: }, 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: { 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: "op": "emit" 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: ] 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: } 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout:] 2023-12-01T22:03:18.386 INFO:tasks.workunit.client.0.smithi170.stdout: 2023-12-01T22:03:18.393 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls 2023-12-01T22:03:19.389 INFO:tasks.workunit.client.0.smithi170.stdout:replicated_rule 2023-12-01T22:03:19.390 INFO:tasks.workunit.client.0.smithi170.stdout:erasure-code 2023-12-01T22:03:19.401 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule list 2023-12-01T22:03:19.781 INFO:tasks.workunit.client.0.smithi170.stdout:replicated_rule 2023-12-01T22:03:19.781 INFO:tasks.workunit.client.0.smithi170.stdout:erasure-code 2023-12-01T22:03:19.793 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule create-simple foo default host 2023-12-01T22:03:20.435 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo already exists 2023-12-01T22:03:20.445 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule create-simple foo default host 2023-12-01T22:03:20.833 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo already exists 2023-12-01T22:03:20.842 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule create-simple bar default host 2023-12-01T22:03:21.353 INFO:tasks.workunit.client.0.smithi170.stderr:rule bar already exists 2023-12-01T22:03:21.363 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm-device-class all 2023-12-01T22:03:23.334 INFO:tasks.workunit.client.0.smithi170.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2023-12-01T22:03:23.346 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush set-device-class ssd osd.0 2023-12-01T22:03:25.833 INFO:tasks.workunit.client.0.smithi170.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2023-12-01T22:03:25.845 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush set-device-class hdd osd.1 2023-12-01T22:03:26.862 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-12-01T22:03:26.862 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-12-01T22:03:26.862 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-01T22:03:26.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-01T22:03:26.865 INFO:tasks.ceph.mon.c:Started 2023-12-01T22:03:26.865 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-12-01T22:03:26.865 INFO:tasks.ceph.mon.a:Sent signal 19 2023-12-01T22:03:26.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-12-01T22:03:32.158 INFO:tasks.workunit.client.0.smithi170.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2023-12-01T22:03:32.168 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2023-12-01T22:03:33.274 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo-ssd already exists 2023-12-01T22:03:33.283 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2023-12-01T22:03:34.285 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo-hdd already exists 2023-12-01T22:03:34.294 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls-by-class ssd 2023-12-01T22:03:34.295 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo-ssd 2023-12-01T22:03:34.688 INFO:tasks.workunit.client.0.smithi170.stdout:foo-ssd 2023-12-01T22:03:34.689 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls-by-class ssd 2023-12-01T22:03:34.689 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false grep foo-hdd 2023-12-01T22:03:34.689 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:34.689 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo-hdd 2023-12-01T22:03:35.081 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:35.081 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls-by-class hdd 2023-12-01T22:03:35.081 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo-hdd 2023-12-01T22:03:36.305 INFO:tasks.workunit.client.0.smithi170.stdout:foo-hdd 2023-12-01T22:03:36.306 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls-by-class hdd 2023-12-01T22:03:36.306 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false grep foo-ssd 2023-12-01T22:03:36.306 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:03:36.306 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo-ssd 2023-12-01T22:03:37.531 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:03:37.531 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2023-12-01T22:03:38.316 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2023-12-01T22:03:40.312 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'ec-foo' already exists 2023-12-01T22:03:40.322 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2023-12-01T22:03:41.311 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'ec-foo' does not exist 2023-12-01T22:03:41.321 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls 2023-12-01T22:03:41.321 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo 2023-12-01T22:03:41.723 INFO:tasks.workunit.client.0.smithi170.stdout:foo 2023-12-01T22:03:41.723 INFO:tasks.workunit.client.0.smithi170.stdout:foo-ssd 2023-12-01T22:03:41.723 INFO:tasks.workunit.client.0.smithi170.stdout:foo-hdd 2023-12-01T22:03:41.723 INFO:tasks.workunit.client.0.smithi170.stdout:ec-foo 2023-12-01T22:03:41.723 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rename foo foo-asdf 2023-12-01T22:03:41.866 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-12-01T22:03:41.866 INFO:tasks.ceph.mon.a:Sent signal 18 2023-12-01T22:03:41.866 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:03:41.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:03:42.827 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:03:42.841 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:03:42.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:03:42.841 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:03:42.942 INFO:tasks.workunit.client.0.smithi170.stderr:already renamed to 'foo-asdf' 2023-12-01T22:03:42.952 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rename foo foo-asdf 2023-12-01T22:03:42.980 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 40, 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:03:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:42.991 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:42.992 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:03:42.993 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:03:42.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 40, 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:43.133 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:43.134 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:43.135 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:43.136 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:43.137 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:43.138 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:43.139 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:43.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.141 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.142 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:43.143 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:43.144 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:03:43.144 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:03:43.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:03:43.285 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:43.285 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:43.285 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:43.285 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:03:43.285 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 40, 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:43.286 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:43.288 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:43.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.290 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.291 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:43.292 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:43.293 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.294 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:43.296 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:43.297 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:03:43.297 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:03:43.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:03:43.346 INFO:tasks.workunit.client.0.smithi170.stderr:already renamed to 'foo-asdf' 2023-12-01T22:03:43.355 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rename bar bar-asdf 2023-12-01T22:03:43.677 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.674+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.677 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.674+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=6,logm=70} crc {auth=255623669,config=3140514009,health=4246560827,kv=2377187526,logm=2632587483}) 2023-12-01T22:03:43.677 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.674+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=7,logm=70} crc {auth=139450136,config=3140514009,health=516918166,kv=2377187526,logm=2216360853}) 2023-12-01T22:03:43.677 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.674+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.677 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.674+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=6,logm=70} crc {auth=255623669,config=3140514009,health=4246560827,kv=2377187526,logm=2632587483}) 2023-12-01T22:03:43.677 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.674+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=1,health=10,kv=7,logm=72} crc {auth=2702512717,config=1228700967,health=1281764270,kv=2377187526,logm=877126260}) 2023-12-01T22:03:43.679 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.677+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.679 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.677+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260131933}) 2023-12-01T22:03:43.679 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.677+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2793399516}) 2023-12-01T22:03:43.679 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.677+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.679 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.677+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260131933}) 2023-12-01T22:03:43.679 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.677+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1318923841}) 2023-12-01T22:03:43.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:03:43.825 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.823+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.826 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.823+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=51} crc {logm=1173482401,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=2197616093,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2060728947,osd_pg_creating=540659982,osdmap=298957343}) 2023-12-01T22:03:43.826 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.823+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=17,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=52} crc {logm=3397952142,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=980248617,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=540659983,osdmap=238087556}) 2023-12-01T22:03:43.826 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.823+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.826 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.823+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=51} crc {logm=1173482401,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=2197616093,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2060728947,osd_pg_creating=540659982,osdmap=298957343}) 2023-12-01T22:03:43.826 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.823+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=18,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=2,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=52} crc {logm=455191567,mdsmap=1531151339,mgr=1268538720,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=2197616094,mon_config_key=3439908406,monmap=342008835,osd_metadata=2309343068,osd_pg_creating=540659982,osdmap=645700962}) 2023-12-01T22:03:43.833 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.830+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.833 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.830+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=50496240}) 2023-12-01T22:03:43.833 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.830+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=15810553}) 2023-12-01T22:03:43.833 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.830+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:03:43.833 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.830+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=50496240}) 2023-12-01T22:03:43.833 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:03:43.830+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=57} crc {osdmap=409451226}) 2023-12-01T22:03:43.954 INFO:tasks.workunit.client.0.smithi170.stderr:already renamed to 'bar-asdf' 2023-12-01T22:03:43.964 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls 2023-12-01T22:03:43.964 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo-asdf 2023-12-01T22:03:44.356 INFO:tasks.workunit.client.0.smithi170.stdout:foo-asdf 2023-12-01T22:03:44.356 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls 2023-12-01T22:03:44.356 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep bar-asdf 2023-12-01T22:03:44.691 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:03:44.691 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:03:44.750 INFO:tasks.workunit.client.0.smithi170.stdout:bar-asdf 2023-12-01T22:03:44.751 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm foo 2023-12-01T22:03:44.751 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 'does not exist' 2023-12-01T22:03:45.080 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:03:45.094 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:03:45.094 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:03:45.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:03:45.095 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:03:45.155 INFO:tasks.workunit.client.0.smithi170.stdout:rule foo does not exist 2023-12-01T22:03:45.156 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm bar 2023-12-01T22:03:45.156 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 'does not exist' 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 40, 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:45.234 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 2, 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.235 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.236 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:45.237 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.238 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:45.240 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:45.241 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.241 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.244 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.245 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.246 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.247 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.248 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:45.248 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:45.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:03:45.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:03:45.249 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 40, 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 2, 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:45.389 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.390 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:45.391 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.392 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.393 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.394 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.395 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.396 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.397 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.398 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:45.398 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.398 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:45.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:45.400 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:45.401 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:03:45.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:03:45.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:03:45.542 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:03:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:03:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 40, 2023-12-01T22:03:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 2, 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:03:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:03:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:03:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:03:45.553 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:03:45.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:03:45.554 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-12-01T22:03:45.554 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-01T22:03:45.554 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-12-01T22:03:45.554 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-01T22:03:45.555 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-01T22:03:45.555 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:03:45.555 INFO:tasks.workunit.client.0.smithi170.stdout:rule bar does not exist 2023-12-01T22:03:45.555 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rename foo-asdf foo 2023-12-01T22:03:45.569 INFO:tasks.ceph.mon.b:Stopped 2023-12-01T22:03:45.570 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:03:45.570 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:03:46.772 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:03:49.789 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:03:50.178 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:03:53.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:04:00.181 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:04:00.194 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:04:00.194 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-01T22:04:00.195 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:04:00.334 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:00.334 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:00.334 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:00.334 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:04:00.334 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 44, 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:00.335 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.336 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.338 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:00.340 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.341 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.342 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.343 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:04:00.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.345 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.345 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:00.345 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:00.346 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:04:00.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-01T22:04:00.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 44, 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:00.486 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:00.487 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:00.488 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:00.489 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:00.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:00.491 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:00.492 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:00.493 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:00.494 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.495 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:00.496 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:00.497 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:00.498 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:04:00.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:04:01.004 INFO:tasks.workunit.client.0.smithi170.stderr:already renamed to 'foo' 2023-12-01T22:04:01.013 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rename foo-asdf foo 2023-12-01T22:04:01.403 INFO:tasks.workunit.client.0.smithi170.stderr:already renamed to 'foo' 2023-12-01T22:04:01.413 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rename bar-asdf bar 2023-12-01T22:04:01.993 INFO:tasks.workunit.client.0.smithi170.stderr:already renamed to 'bar' 2023-12-01T22:04:02.003 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls 2023-12-01T22:04:02.003 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false grep foo-asdf 2023-12-01T22:04:02.003 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:02.003 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo-asdf 2023-12-01T22:04:02.400 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:02.400 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule ls 2023-12-01T22:04:02.400 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false grep bar-asdf 2023-12-01T22:04:02.400 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:02.400 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep bar-asdf 2023-12-01T22:04:03.414 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:03.414 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm foo 2023-12-01T22:04:04.026 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo does not exist 2023-12-01T22:04:04.036 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm foo 2023-12-01T22:04:04.425 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo does not exist 2023-12-01T22:04:04.434 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm bar 2023-12-01T22:04:05.032 INFO:tasks.workunit.client.0.smithi170.stderr:rule bar does not exist 2023-12-01T22:04:05.042 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create pinning_pool 1 2023-12-01T22:04:06.032 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'pinning_pool' already exists 2023-12-01T22:04:06.042 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2023-12-01T22:04:06.042 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:06.043 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm replicated_rule 2023-12-01T22:04:06.322 INFO:tasks.workunit.client.0.smithi170.stderr:Error EBUSY: crush ruleset replicated_rule 0 is in use 2023-12-01T22:04:06.323 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:06.323 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2023-12-01T22:04:07.990 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'pinning_pool' does not exist 2023-12-01T22:04:08.000 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush add-bucket foo osd 2023-12-01T22:04:08.000 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:08.000 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add-bucket foo osd 2023-12-01T22:04:08.277 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2023-12-01T22:04:08.279 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:08.279 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add-bucket foo root 2023-12-01T22:04:10.868 INFO:tasks.workunit.client.0.smithi170.stderr:bucket 'foo' already exists 2023-12-01T22:04:10.879 INFO:tasks.workunit.client.0.smithi170.stderr:++ ceph osd create 2023-12-01T22:04:11.897 INFO:tasks.workunit.client.0.smithi170.stderr:+ o1=7 2023-12-01T22:04:11.897 INFO:tasks.workunit.client.0.smithi170.stderr:++ ceph osd create 2023-12-01T22:04:13.922 INFO:tasks.workunit.client.0.smithi170.stderr:+ o2=9 2023-12-01T22:04:13.923 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2023-12-01T22:04:15.017 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2023-12-01T22:04:15.027 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2023-12-01T22:04:15.418 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2023-12-01T22:04:15.428 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2023-12-01T22:04:16.994 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2023-12-01T22:04:17.004 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2023-12-01T22:04:17.395 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2023-12-01T22:04:17.405 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add-bucket bar root 2023-12-01T22:04:18.045 INFO:tasks.workunit.client.0.smithi170.stderr:bucket 'bar' already exists 2023-12-01T22:04:18.054 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add-bucket bar root 2023-12-01T22:04:18.445 INFO:tasks.workunit.client.0.smithi170.stderr:bucket 'bar' already exists 2023-12-01T22:04:18.455 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush link host1 root=bar 2023-12-01T22:04:19.059 INFO:tasks.workunit.client.0.smithi170.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2023-12-01T22:04:19.070 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush link host1 root=bar 2023-12-01T22:04:19.462 INFO:tasks.workunit.client.0.smithi170.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2023-12-01T22:04:19.471 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush link host2 root=bar 2023-12-01T22:04:20.065 INFO:tasks.workunit.client.0.smithi170.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2023-12-01T22:04:20.075 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush link host2 root=bar 2023-12-01T22:04:20.465 INFO:tasks.workunit.client.0.smithi170.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2023-12-01T22:04:20.476 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:20.476 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c osd.7 2023-12-01T22:04:20.476 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 2 2023-12-01T22:04:20.501 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-01T22:04:20.501 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-12-01T22:04:20.501 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-01T22:04:20.501 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-01T22:04:20.503 INFO:tasks.ceph.mon.b:Started 2023-12-01T22:04:20.503 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:04:20.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:04:20.870 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:20.871 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c host1 2023-12-01T22:04:20.871 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 2 2023-12-01T22:04:20.891 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:04:21.259 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:21.260 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c osd.9 2023-12-01T22:04:21.260 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 2 2023-12-01T22:04:21.653 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:21.653 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c host2 2023-12-01T22:04:21.653 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 2 2023-12-01T22:04:22.043 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush rm host1 foo 2023-12-01T22:04:22.043 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:22.043 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm host1 foo 2023-12-01T22:04:22.323 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOTEMPTY: (39) Directory not empty 2023-12-01T22:04:22.325 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:22.325 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush unlink host1 foo 2023-12-01T22:04:23.086 INFO:tasks.workunit.client.0.smithi170.stderr:item -11 does not appear in that position 2023-12-01T22:04:23.095 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush unlink host1 foo 2023-12-01T22:04:23.495 INFO:tasks.workunit.client.0.smithi170.stderr:item -11 does not appear in that position 2023-12-01T22:04:23.505 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:23.506 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c host1 2023-12-01T22:04:23.506 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 1 2023-12-01T22:04:23.895 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush rm foo 2023-12-01T22:04:23.895 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:23.895 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm foo 2023-12-01T22:04:23.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:04:24.175 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOTEMPTY: (39) Directory not empty 2023-12-01T22:04:24.177 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:24.177 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush rm bar 2023-12-01T22:04:24.177 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:24.177 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm bar 2023-12-01T22:04:24.297 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:04:24.455 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOTEMPTY: (39) Directory not empty 2023-12-01T22:04:24.457 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:24.457 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush unlink host1 bar 2023-12-01T22:04:25.103 INFO:tasks.workunit.client.0.smithi170.stderr:item -11 does not appear in that position 2023-12-01T22:04:25.114 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:25.114 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c host1 2023-12-01T22:04:25.114 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 1 2023-12-01T22:04:25.506 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.7 host1 2023-12-01T22:04:26.145 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.7' does not appear in the crush map 2023-12-01T22:04:26.155 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm host1 2023-12-01T22:04:27.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:04:27.338 INFO:tasks.workunit.client.0.smithi170.stderr:device 'host1' does not appear in the crush map 2023-12-01T22:04:27.348 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:27.348 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -c host1 2023-12-01T22:04:27.348 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep -q 0 2023-12-01T22:04:27.701 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:04:27.715 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:04:27.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:04:27.716 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:04:27.739 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd tree-from host1 2023-12-01T22:04:27.739 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:27.740 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree-from host1 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 46, 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:04:27.859 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:27.860 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:27.861 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:27.862 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:27.863 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:27.864 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:27.865 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:27.866 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:27.867 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.868 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.869 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:27.870 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:27.871 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:27.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:04:27.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:04:27.872 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:04:28.014 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOENT: bucket 'host1' does not exist 2023-12-01T22:04:28.014 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:28.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:28.014 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:28.014 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:04:28.014 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 46, 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:28.015 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:28.016 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:28.017 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:28.018 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:28.018 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:28.018 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:28.018 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:28.018 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:28.018 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:28.019 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:28.019 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:28.019 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:28.019 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.020 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:28.022 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:28.023 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:28.024 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.026 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:28.027 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:28.028 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.029 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.029 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:28.029 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:28.031 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:28.031 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree-from host2 2023-12-01T22:04:28.031 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:04:28.032 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:04:28.032 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 46, 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:28.173 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.174 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:28.175 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:28.176 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:28.177 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.178 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:28.179 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.180 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:28.181 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:28.182 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:28.183 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:04:28.184 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:28.184 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:28.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:28.184 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:28.184 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:28.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:04:28.185 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:04:28.185 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:04:28.395 INFO:tasks.workunit.client.0.smithi170.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-12-01T22:04:28.395 INFO:tasks.workunit.client.0.smithi170.stdout:-16 1.00000 host host2 2023-12-01T22:04:28.395 INFO:tasks.workunit.client.0.smithi170.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2023-12-01T22:04:28.407 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd tree-from osd.9 2023-12-01T22:04:28.407 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:28.408 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree-from osd.9 2023-12-01T22:04:28.562 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.561+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.563 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.561+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=3276320793,config=3140514009,health=3619357427,kv=2377187526,logm=2379758042}) 2023-12-01T22:04:28.563 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.561+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=3276320793,config=3140514009,health=3619357427,kv=2377187526,logm=2219978700}) 2023-12-01T22:04:28.563 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.561+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.563 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.561+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=3276320793,config=3140514009,health=3619357427,kv=2377187526,logm=2379758042}) 2023-12-01T22:04:28.563 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.561+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=7,logm=69} crc {auth=1401501983,config=3140514009,health=3619357427,kv=2377187526,logm=2330441112}) 2023-12-01T22:04:28.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:04:28.606 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.604+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.606 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.604+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488443475}) 2023-12-01T22:04:28.606 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.604+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1952195681}) 2023-12-01T22:04:28.606 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.604+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.606 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.604+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488443475}) 2023-12-01T22:04:28.606 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.604+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516878453}) 2023-12-01T22:04:28.682 INFO:tasks.workunit.client.0.smithi170.stderr:Error EINVAL: "osd.9" is not a bucket 2023-12-01T22:04:28.684 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:28.684 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush rm bar 2023-12-01T22:04:28.685 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:28.685 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm bar 2023-12-01T22:04:28.857 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.855+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.857 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.855+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=1098845760,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=321249796,mon_config_key=161422294}) 2023-12-01T22:04:28.857 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.855+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=3,monmap=3} crc {logm=1628506697,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=493165042,mon_config_key=2250838263,monmap=3233185808}) 2023-12-01T22:04:28.857 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.855+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.857 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.855+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=1098845760,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=321249796,mon_config_key=161422294}) 2023-12-01T22:04:28.857 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.855+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=3,monmap=1} crc {logm=2046841967,mdsmap=1531151339,mgr=1268538720,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=321249796,mon_config_key=2250838263,monmap=2599872377}) 2023-12-01T22:04:28.858 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.856+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.858 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.856+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=90} crc {mon_config_key=1711957015,monmap=3233185808,osd_metadata=2060728947,osd_pg_creating=2251434291,osdmap=2082210852}) 2023-12-01T22:04:28.858 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.856+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=89} crc {mon_config_key=1711957015,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=2251434291,osdmap=3377488174}) 2023-12-01T22:04:28.858 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.856+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.859 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.856+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=90} crc {mon_config_key=1711957015,monmap=3233185808,osd_metadata=2060728947,osd_pg_creating=2251434291,osdmap=2082210852}) 2023-12-01T22:04:28.859 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.856+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=1,monmap=2,osd_metadata=5,osd_pg_creating=1,osdmap=91} crc {mon_config_key=1711957015,monmap=342008835,osd_metadata=565211891,osd_pg_creating=2251434291,osdmap=329837409}) 2023-12-01T22:04:28.861 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.860+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.861 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.860+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=2850592144}) 2023-12-01T22:04:28.861 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.860+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=1964008291}) 2023-12-01T22:04:28.862 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.860+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:04:28.862 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.860+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=2850592144}) 2023-12-01T22:04:28.862 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:04:28.860+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=3206894613}) 2023-12-01T22:04:28.965 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOTEMPTY: (39) Directory not empty 2023-12-01T22:04:28.967 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:28.967 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush unlink host2 2023-12-01T22:04:29.577 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:04:29.578 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:04:29.960 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:04:29.974 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:04:29.974 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:04:29.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:04:29.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 46, 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:30.114 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:30.115 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:30.116 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.117 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.121 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:30.122 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:30.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:30.124 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:30.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:04:30.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:04:30.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:04:30.281 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:30.281 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:30.281 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:30.281 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:04:30.281 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 46, 2023-12-01T22:04:30.281 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.282 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:30.284 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:30.285 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.286 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.288 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:30.289 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.291 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:30.292 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:30.293 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:04:30.293 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:04:30.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:04:30.434 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 46, 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 4, 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:30.435 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.436 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:30.437 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:30.438 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.440 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.442 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.443 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.444 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:30.445 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:30.446 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:04:30.446 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-12-01T22:04:30.446 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2023-12-01T22:04:30.447 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-12-01T22:04:30.447 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-01T22:04:30.447 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-01T22:04:30.447 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:04:30.457 INFO:tasks.ceph.mon.b:Stopped 2023-12-01T22:04:30.457 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-12-01T22:04:30.457 INFO:tasks.ceph.mon.a:Sent signal 19 2023-12-01T22:04:30.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-12-01T22:04:45.458 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-12-01T22:04:45.459 INFO:tasks.ceph.mon.a:Sent signal 18 2023-12-01T22:04:45.459 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:04:45.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:04:50.773 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":50,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:04:50.787 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:04:50.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-01T22:04:50.787 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:04:50.927 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:50.927 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:50.927 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 50, 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:50.928 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:50.929 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:50.930 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:50.931 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:50.932 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:50.933 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.934 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:50.935 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:50.936 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:50.937 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:50.938 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:50.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:50.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:50.941 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:50.941 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:50.942 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:04:50.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-01T22:04:50.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:04:51.081 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:04:51.081 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:51.081 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:51.081 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 50, 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:51.082 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:51.083 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:04:51.084 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:04:51.085 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:04:51.086 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:51.088 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:04:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.090 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:51.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.091 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:04:51.091 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.092 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.093 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:04:51.094 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:04:51.095 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:04:51.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:04:51.511 INFO:tasks.workunit.client.0.smithi170.stderr:removed item id -16 name 'host2' from crush map 2023-12-01T22:04:51.523 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2023-12-01T22:04:52.628 INFO:tasks.workunit.client.0.smithi170.stderr:bucket 'host-for-test' already exists 2023-12-01T22:04:52.638 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:52.638 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep host-for-test 2023-12-01T22:04:53.024 INFO:tasks.workunit.client.0.smithi170.stdout:-11 0 host host-for-test 2023-12-01T22:04:53.025 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:53.025 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep rack-for-test 2023-12-01T22:04:53.416 INFO:tasks.workunit.client.0.smithi170.stdout:-12 0 rack rack-for-test 2023-12-01T22:04:53.417 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:04:53.417 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep root-for-test 2023-12-01T22:04:53.809 INFO:tasks.workunit.client.0.smithi170.stdout:-14 0 root root-for-test 2023-12-01T22:04:53.809 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm host-for-test 2023-12-01T22:04:54.662 INFO:tasks.workunit.client.0.smithi170.stderr:device 'host-for-test' does not appear in the crush map 2023-12-01T22:04:54.672 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm rack-for-test 2023-12-01T22:04:55.664 INFO:tasks.workunit.client.0.smithi170.stderr:device 'rack-for-test' does not appear in the crush map 2023-12-01T22:04:55.674 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm root-for-test 2023-12-01T22:04:57.679 INFO:tasks.workunit.client.0.smithi170.stderr:device 'root-for-test' does not appear in the crush map 2023-12-01T22:04:57.689 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2023-12-01T22:04:59.687 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo-rule already exists 2023-12-01T22:04:59.697 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush rm foo 2023-12-01T22:04:59.697 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:04:59.697 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm foo 2023-12-01T22:04:59.974 INFO:tasks.workunit.client.0.smithi170.stderr:Error EBUSY: (16) Device or resource busy 2023-12-01T22:04:59.976 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:04:59.976 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rule rm foo-rule 2023-12-01T22:05:00.584 INFO:tasks.workunit.client.0.smithi170.stderr:rule foo-rule does not exist 2023-12-01T22:05:00.594 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm bar 2023-12-01T22:05:01.693 INFO:tasks.workunit.client.0.smithi170.stderr:device 'bar' does not appear in the crush map 2023-12-01T22:05:01.703 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm foo 2023-12-01T22:05:03.711 INFO:tasks.workunit.client.0.smithi170.stderr:device 'foo' does not appear in the crush map 2023-12-01T22:05:03.720 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.9 host2 2023-12-01T22:05:04.714 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.9' does not appear in the crush map 2023-12-01T22:05:04.723 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm host2 2023-12-01T22:05:06.732 INFO:tasks.workunit.client.0.smithi170.stderr:device 'host2' does not appear in the crush map 2023-12-01T22:05:06.742 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add-bucket foo host 2023-12-01T22:05:07.757 INFO:tasks.workunit.client.0.smithi170.stderr:bucket 'foo' already exists 2023-12-01T22:05:07.767 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush move foo root=default rack=localrack 2023-12-01T22:05:09.361 INFO:tasks.workunit.client.0.smithi170.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2023-12-01T22:05:09.371 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2023-12-01T22:05:10.371 INFO:tasks.workunit.client.0.smithi170.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2023-12-01T22:05:10.380 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush move osd.7 host=foo 2023-12-01T22:05:11.096 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-01T22:05:11.096 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-12-01T22:05:11.096 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-01T22:05:11.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-01T22:05:11.099 INFO:tasks.ceph.mon.b:Started 2023-12-01T22:05:11.099 INFO:tasks.mon_thrash:Sending STOP to mon a 2023-12-01T22:05:11.099 INFO:tasks.ceph.mon.a:Sent signal 19 2023-12-01T22:05:11.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-12-01T22:05:22.652 INFO:tasks.workunit.client.0.smithi170.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2023-12-01T22:05:22.662 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd find osd.7 2023-12-01T22:05:22.662 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep host 2023-12-01T22:05:22.662 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep foo 2023-12-01T22:05:23.049 INFO:tasks.workunit.client.0.smithi170.stdout: "host": "foo", 2023-12-01T22:05:23.050 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.7 2023-12-01T22:05:23.349 INFO:tasks.workunit.client.0.smithi170.stderr:2023-12-01T22:05:23.348+0000 7f42eeffd700 0 --2- 172.21.15.170:0/2570302745 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f42f0099fc0 0x7f42f0129ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-01T22:05:24.646 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.7' does not appear in the crush map 2023-12-01T22:05:24.656 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.9 2023-12-01T22:05:25.665 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.9' does not appear in the crush map 2023-12-01T22:05:25.675 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm foo 2023-12-01T22:05:26.100 INFO:tasks.mon_thrash:Sending CONT to mon a 2023-12-01T22:05:26.100 INFO:tasks.ceph.mon.a:Sent signal 18 2023-12-01T22:05:26.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:05:26.101 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:05:26.497 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:05:26.511 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:05:26.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:05:26.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:05:26.650 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:26.650 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:26.650 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:26.650 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:05:26.650 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 58, 2023-12-01T22:05:26.650 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.651 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.652 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:26.653 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:26.654 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.655 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.657 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.658 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.659 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.660 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.661 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:26.662 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:26.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:05:26.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:05:26.664 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 58, 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.804 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.805 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:26.820 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.821 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.821 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:26.821 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:26.823 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:05:26.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:05:26.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:05:26.962 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 58, 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:26.963 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.964 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:26.965 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:26.966 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.967 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.968 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:26.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:26.973 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:26.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:05:26.974 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:05:26.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:05:27.311 INFO:tasks.workunit.client.0.smithi170.stderr:device 'foo' does not appear in the crush map 2023-12-01T22:05:27.321 INFO:tasks.workunit.client.0.smithi170.stderr:++ ceph osd create 2023-12-01T22:05:27.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.350+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=1944225238,config=3140514009,health=2718883078,kv=2377187526,logm=4196680291}) 2023-12-01T22:05:27.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.351+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=3230137845,config=3140514009,health=2718883078,kv=2377187526,logm=3096238876}) 2023-12-01T22:05:27.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.351+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.353 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.351+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=1944225238,config=3140514009,health=2718883078,kv=2377187526,logm=4196680291}) 2023-12-01T22:05:27.354 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.351+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=10,kv=7,logm=70} crc {auth=3935070628,config=3140514009,health=4072375402,kv=2377187526,logm=2869276417}) 2023-12-01T22:05:27.354 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163868428}) 2023-12-01T22:05:27.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2693644356}) 2023-12-01T22:05:27.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163868428}) 2023-12-01T22:05:27.355 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.352+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658930674}) 2023-12-01T22:05:27.357 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.355+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.357 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.355+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952430334}) 2023-12-01T22:05:27.358 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.355+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570106311}) 2023-12-01T22:05:27.358 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.355+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.358 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.355+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952430334}) 2023-12-01T22:05:27.358 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.355+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986661900}) 2023-12-01T22:05:27.360 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.357+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.360 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.357+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=43} crc {logm=2906181688,mdsmap=1531151339,mgr=2165843201,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3892291207,mon_config_key=2250838263,monmap=342008835,osd_metadata=2309343068,osd_pg_creating=2511244211,osdmap=702883582}) 2023-12-01T22:05:27.360 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.357+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=44} crc {logm=2768192053,mdsmap=1531151339,mgr=1977267238,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3892291207,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=2511244211,osdmap=3717389642}) 2023-12-01T22:05:27.360 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.357+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.360 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.357+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=43} crc {logm=2906181688,mdsmap=1531151339,mgr=2165843201,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3892291207,mon_config_key=2250838263,monmap=342008835,osd_metadata=2309343068,osd_pg_creating=2511244211,osdmap=702883582}) 2023-12-01T22:05:27.360 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.357+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=23,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=2,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=44} crc {logm=2906181688,mdsmap=1531151339,mgr=1977267238,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=3892291207,mon_config_key=161422294,monmap=3233185808,osd_metadata=3616433184,osd_pg_creating=2511244211,osdmap=957187815}) 2023-12-01T22:05:27.361 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.358+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.361 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.358+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1867744534}) 2023-12-01T22:05:27.361 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.358+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2740727543}) 2023-12-01T22:05:27.361 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.358+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.361 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.358+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1867744534}) 2023-12-01T22:05:27.361 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.358+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2111770175}) 2023-12-01T22:05:27.366 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.364+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.366 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.364+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=10} crc {osdmap=603839567}) 2023-12-01T22:05:27.366 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.364+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=11} crc {osdmap=1091386822}) 2023-12-01T22:05:27.366 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.364+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:05:27.366 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.364+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=10} crc {osdmap=603839567}) 2023-12-01T22:05:27.366 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:05:27.364+0000 7ff2fde71700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=421640693}) 2023-12-01T22:05:27.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:05:28.373 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:05:28.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:05:28.751 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:05:28.765 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:05:28.765 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:05:28.765 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:05:28.765 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 58, 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:28.905 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 2, 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:28.906 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.907 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:28.908 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:28.909 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:28.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.913 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:28.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:28.915 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:28.916 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:28.917 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:28.917 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:28.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:28.917 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:28.917 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:28.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:05:28.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:05:28.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:05:29.058 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:29.058 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:29.058 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:29.058 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:05:29.058 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 58, 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 2, 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:29.059 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:29.060 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:29.061 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:29.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.063 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:29.065 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:29.067 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.068 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:29.069 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:29.070 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:05:29.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:05:29.070 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:05:29.231 INFO:tasks.workunit.client.0.smithi170.stderr:+ o3=11 2023-12-01T22:05:29.231 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 11 123 root=default 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 58, 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:05:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:29.424 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:29.425 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:05:29.425 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-12-01T22:05:29.426 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-01T22:05:29.426 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-12-01T22:05:29.426 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-12-01T22:05:29.426 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-01T22:05:29.426 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:05:29.438 INFO:tasks.ceph.mon.a:Stopped 2023-12-01T22:05:29.438 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:05:29.438 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:05:29.848 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:05:32.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:05:33.252 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:05:36.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:05:36.650 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:05:39.665 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:05:44.422 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2023-12-01T22:05:44.432 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:05:44.432 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep osd.11 2023-12-01T22:05:44.432 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 123 2023-12-01T22:05:44.517 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:05:44.531 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:05:44.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-01T22:05:44.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:05:44.673 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:44.673 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:44.673 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:44.673 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:05:44.673 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 60, 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:44.674 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:44.675 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:44.676 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.677 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:44.678 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:44.679 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:44.680 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.682 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.683 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:44.684 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:44.685 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:05:44.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-12-01T22:05:44.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:05:44.821 INFO:tasks.workunit.client.0.smithi170.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2023-12-01T22:05:44.821 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush reweight osd.11 113 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 60, 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:05:44.826 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:44.827 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:44.828 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:05:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:05:44.830 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:44.831 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:44.832 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:44.834 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:05:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.837 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:05:44.838 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:05:44.839 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:05:44.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:05:44.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:05:46.316 INFO:tasks.workunit.client.0.smithi170.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2023-12-01T22:05:46.327 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2023-12-01T22:05:46.328 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:05:46.328 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush reweight osd.11 123456 2023-12-01T22:05:46.611 INFO:tasks.workunit.client.0.smithi170.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2023-12-01T22:05:46.613 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:05:46.614 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:05:46.614 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep osd.11 2023-12-01T22:05:46.614 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 113 2023-12-01T22:05:46.999 INFO:tasks.workunit.client.0.smithi170.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2023-12-01T22:05:46.999 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.11 2023-12-01T22:05:48.260 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.11' does not appear in the crush map 2023-12-01T22:05:48.270 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd rm osd.11 2023-12-01T22:05:48.554 INFO:tasks.ceph.mon.b.smithi170.stderr:2023-12-01T22:05:48.552+0000 7fcf82369700 -1 mon.b@1(leader).osd e114 definitely_dead 0 2023-12-01T22:05:49.440 INFO:tasks.ceph.mon.b.smithi170.stderr:2023-12-01T22:05:49.438+0000 7fcf82369700 -1 mon.b@1(leader).osd e115 definitely_dead 0 2023-12-01T22:05:49.442 INFO:tasks.workunit.client.0.smithi170.stderr:osd.11 does not exist. 2023-12-01T22:05:49.452 INFO:tasks.workunit.client.0.smithi170.stderr:++ ceph osd create 2023-12-01T22:05:51.201 INFO:tasks.workunit.client.0.smithi170.stderr:+ o4=12 2023-12-01T22:05:51.202 INFO:tasks.workunit.client.0.smithi170.stderr:++ ceph osd create 2023-12-01T22:05:53.368 INFO:tasks.workunit.client.0.smithi170.stderr:+ o5=14 2023-12-01T22:05:53.368 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2023-12-01T22:05:55.300 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2023-12-01T22:05:55.310 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2023-12-01T22:05:56.312 INFO:tasks.workunit.client.0.smithi170.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2023-12-01T22:05:56.322 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:05:56.322 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep osd.12 2023-12-01T22:05:56.322 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 123 2023-12-01T22:05:56.714 INFO:tasks.workunit.client.0.smithi170.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2023-12-01T22:05:56.715 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:05:56.715 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep osd.14 2023-12-01T22:05:56.715 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 123 2023-12-01T22:05:57.103 INFO:tasks.workunit.client.0.smithi170.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2023-12-01T22:05:57.103 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush reweight-subtree foobaz 155 2023-12-01T22:05:59.395 INFO:tasks.workunit.client.0.smithi170.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2023-12-01T22:05:59.406 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2023-12-01T22:05:59.406 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:05:59.407 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2023-12-01T22:05:59.693 INFO:tasks.workunit.client.0.smithi170.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2023-12-01T22:05:59.695 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:05:59.695 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:05:59.695 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep osd.12 2023-12-01T22:05:59.695 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 155 2023-12-01T22:06:00.084 INFO:tasks.workunit.client.0.smithi170.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2023-12-01T22:06:00.085 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd tree 2023-12-01T22:06:00.085 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep osd.14 2023-12-01T22:06:00.085 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep 155 2023-12-01T22:06:00.471 INFO:tasks.workunit.client.0.smithi170.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2023-12-01T22:06:00.472 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.12 2023-12-01T22:06:02.308 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.12' does not appear in the crush map 2023-12-01T22:06:02.318 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush rm osd.14 2023-12-01T22:06:03.942 INFO:tasks.workunit.client.0.smithi170.stderr:device 'osd.14' does not appear in the crush map 2023-12-01T22:06:03.952 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd rm osd.12 2023-12-01T22:06:04.232 INFO:tasks.ceph.mon.b.smithi170.stderr:2023-12-01T22:06:04.229+0000 7fcf82369700 -1 mon.b@1(leader).osd e125 definitely_dead 0 2023-12-01T22:06:04.841 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-12-01T22:06:04.841 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-12-01T22:06:04.842 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-01T22:06:04.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-01T22:06:04.844 INFO:tasks.ceph.mon.a:Started 2023-12-01T22:06:04.844 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:06:04.845 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:04.941 INFO:tasks.ceph.mon.b.smithi170.stderr:2023-12-01T22:06:04.939+0000 7fcf82369700 -1 mon.b@1(leader).osd e126 definitely_dead 0 2023-12-01T22:06:04.944 INFO:tasks.workunit.client.0.smithi170.stderr:osd.12 does not exist. 2023-12-01T22:06:04.953 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd rm osd.14 2023-12-01T22:06:05.233 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:05.238 INFO:tasks.ceph.mon.b.smithi170.stderr:2023-12-01T22:06:05.237+0000 7fcf82369700 -1 mon.b@1(leader).osd e126 definitely_dead 0 2023-12-01T22:06:05.945 INFO:tasks.ceph.mon.b.smithi170.stderr:2023-12-01T22:06:05.944+0000 7fcf82369700 -1 mon.b@1(leader).osd e127 definitely_dead 0 2023-12-01T22:06:05.948 INFO:tasks.workunit.client.0.smithi170.stderr:osd.14 does not exist. 2023-12-01T22:06:05.957 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd set-require-min-compat-client luminous 2023-12-01T22:06:08.206 INFO:tasks.workunit.client.0.smithi170.stderr:set require_min_compat_client to luminous 2023-12-01T22:06:08.218 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set dump 2023-12-01T22:06:08.248 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:08.617 INFO:tasks.workunit.client.0.smithi170.stdout:{} 2023-12-01T22:06:08.629 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set ls 2023-12-01T22:06:08.635 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:09.032 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2023-12-01T22:06:09.032 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:06:09.032 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2023-12-01T22:06:09.319 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOENT: pool 'fooset' not found 2023-12-01T22:06:09.321 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:06:09.321 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create fooset 8 2023-12-01T22:06:10.320 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'fooset' already exists 2023-12-01T22:06:10.330 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool create barset 8 2023-12-01T22:06:11.596 INFO:tasks.workunit.client.0.smithi170.stderr:pool 'barset' already exists 2023-12-01T22:06:11.604 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd pool set barset size 3 2023-12-01T22:06:11.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:12.036 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:12.050 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:06:12.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:06:12.051 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:06:12.190 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 64, 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 0, 2023-12-01T22:06:12.191 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.192 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.194 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:12.196 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:12.197 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.198 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:12.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:12.200 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:12.202 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:12.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:06:12.202 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 64, 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.342 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:12.343 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:12.344 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.345 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.346 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:12.347 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.348 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.349 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:12.350 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.351 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:12.352 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:12.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:12.354 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:06:12.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 64, 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:12.500 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.501 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:12.504 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:12.505 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.506 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:12.508 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:12.509 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:12.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:12.511 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:12.511 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:12.512 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:12.512 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:06:12.512 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:06:13.503 INFO:tasks.workunit.client.0.smithi170.stderr:set pool 10 size to 3 2023-12-01T22:06:13.514 INFO:tasks.workunit.client.0.smithi170.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2023-12-01T22:06:13.514 INFO:tasks.workunit.client.0.smithi170.stderr:+ set -x 2023-12-01T22:06:13.514 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2023-12-01T22:06:13.716 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.714+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.716 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.714+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=7,logm=66} crc {auth=2115987808,config=3140514009,health=3359746853,kv=2377187526,logm=461907604}) 2023-12-01T22:06:13.716 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.714+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=1,health=11,kv=7,logm=68} crc {auth=2115987808,config=1228700967,health=3609604190,kv=2377187526,logm=4082300225}) 2023-12-01T22:06:13.717 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.714+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.717 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.714+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=7,logm=66} crc {auth=2115987808,config=3140514009,health=3359746853,kv=2377187526,logm=461907604}) 2023-12-01T22:06:13.717 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.714+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=7,logm=66} crc {auth=2115987808,config=3140514009,health=3359746853,kv=2377187526,logm=2975800887}) 2023-12-01T22:06:13.717 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.715+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.717 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.715+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567782323}) 2023-12-01T22:06:13.717 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.715+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387209293}) 2023-12-01T22:06:13.718 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.715+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.718 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.715+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567782323}) 2023-12-01T22:06:13.718 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.715+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1132142967}) 2023-12-01T22:06:13.720 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.717+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.720 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.718+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215072241}) 2023-12-01T22:06:13.720 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.718+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2666467818}) 2023-12-01T22:06:13.721 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.718+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.721 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.718+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215072241}) 2023-12-01T22:06:13.721 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.718+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2255459845}) 2023-12-01T22:06:13.722 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.720+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.722 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.720+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=3197395685,mdsmap=1531151339,mgr=2835230805}) 2023-12-01T22:06:13.723 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.720+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2723407570,mdsmap=1531151339,mgr=305193480}) 2023-12-01T22:06:13.723 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.720+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.723 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.720+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=3197395685,mdsmap=1531151339,mgr=2835230805}) 2023-12-01T22:06:13.723 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.720+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=811548625,mdsmap=1531151339,mgr=814570151}) 2023-12-01T22:06:13.724 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.721+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.724 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.721+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=69} crc {mgr=193308824,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1988309338,mon_config_key=2250838263,monmap=3140514009,osd_metadata=2309343068,osd_pg_creating=4069555348,osdmap=4265137631}) 2023-12-01T22:06:13.724 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.721+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=68} crc {mgr=4266853807,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1988309338,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=4069555348,osdmap=3791578336}) 2023-12-01T22:06:13.724 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.721+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.724 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.721+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=69} crc {mgr=193308824,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1988309338,mon_config_key=2250838263,monmap=3140514009,osd_metadata=2309343068,osd_pg_creating=4069555348,osdmap=4265137631}) 2023-12-01T22:06:13.724 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.721+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=69} crc {mgr=4266853807,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1988309338,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2060728947,osd_pg_creating=4069555348,osdmap=2606248830}) 2023-12-01T22:06:13.726 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.722+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.726 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.723+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2082750488}) 2023-12-01T22:06:13.726 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.723+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1067892861}) 2023-12-01T22:06:13.726 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.723+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.726 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.723+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2082750488}) 2023-12-01T22:06:13.726 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.723+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=883447151}) 2023-12-01T22:06:13.729 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.727+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.730 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.727+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=3106463413}) 2023-12-01T22:06:13.730 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.727+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=639684468}) 2023-12-01T22:06:13.730 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.727+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:06:13.730 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.727+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=3106463413}) 2023-12-01T22:06:13.730 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:06:13.727+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=17} crc {osdmap=2941232501}) 2023-12-01T22:06:13.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:06:14.630 INFO:tasks.workunit.client.0.smithi170.stderr:Error ENOENT: no weight-set for pool 'fooset' 2023-12-01T22:06:14.631 INFO:tasks.workunit.client.0.smithi170.stderr:+ return 0 2023-12-01T22:06:14.632 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set create fooset flat 2023-12-01T22:06:14.733 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:06:14.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:15.074 INFO:tasks.workunit.client.0.smithi170.stderr:weight-set for pool 'fooset' already created 2023-12-01T22:06:15.083 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set create barset positional 2023-12-01T22:06:15.123 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:15.136 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:06:15.137 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-12-01T22:06:15.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:06:15.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:06:15.276 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 64, 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 4, 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:15.286 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:15.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:15.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:06:15.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 64, 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:15.429 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 4, 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.430 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:15.431 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.432 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.434 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:15.435 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.436 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.437 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.438 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:15.439 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:15.441 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:15.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:06:15.441 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 64, 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 4, 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:15.593 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:15.594 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:15.594 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-12-01T22:06:15.594 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-12-01T22:06:15.594 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-12-01T22:06:15.594 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-01T22:06:15.594 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-01T22:06:15.595 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:06:15.606 INFO:tasks.ceph.mon.b:Stopped 2023-12-01T22:06:15.606 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-12-01T22:06:15.607 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:15.726 INFO:tasks.workunit.client.0.smithi170.stderr:weight-set for pool 'barset' already created 2023-12-01T22:06:15.736 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set ls 2023-12-01T22:06:15.737 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep fooset 2023-12-01T22:06:15.981 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:16.145 INFO:tasks.workunit.client.0.smithi170.stdout:fooset 2023-12-01T22:06:16.145 INFO:tasks.workunit.client.0.smithi170.stderr:+ ceph osd crush weight-set ls 2023-12-01T22:06:16.145 INFO:tasks.workunit.client.0.smithi170.stderr:+ grep barset 2023-12-01T22:06:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:06:16.554 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2023-12-01T22:06:16.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-01T22:06:16.814 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/crush_ops.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh' 2023-12-01T22:06:16.883 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ae640b47502d4af0a4472ba651a67d69 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/crush_ops.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh' 2023-12-01T22:06:16.885 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2023-12-01T22:06:16.895 INFO:tasks.mon_thrash:joining mon_thrasher 2023-12-01T22:06:18.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:19.384 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:22.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:22.788 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:25.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:32.198 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:32.211 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-12-01T22:06:32.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-12-01T22:06:32.212 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:06:32.351 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 68, 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:32.352 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:32.353 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.354 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:32.355 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:32.356 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:32.358 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:32.360 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.361 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:32.362 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.363 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:32.364 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:32.364 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.364 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.364 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.364 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:32.365 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:32.365 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:32.365 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:32.365 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:32.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.366 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:32.366 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:32.366 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.367 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:32.368 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:32.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:32.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.371 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.372 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 2 2023-12-01T22:06:32.372 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.372 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.372 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:32.372 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:32.373 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:06:32.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-12-01T22:06:32.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:06:32.513 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:06:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 68, 2023-12-01T22:06:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 1, 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:32.517 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:32.518 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:32.519 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:32.520 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:32.521 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:32.521 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:32.521 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:32.521 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.521 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:32.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.522 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.523 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:32.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:32.525 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.526 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:32.528 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.531 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:32.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:32.533 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:32.534 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:32.534 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:32.534 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:32.534 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:32.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:32.534 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:32.537 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:32.538 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-12-01T22:06:32.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-12-01T22:06:52.539 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-12-01T22:06:52.540 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-12-01T22:06:52.540 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-01T22:06:52.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-01T22:06:52.543 INFO:tasks.ceph.mon.b:Started 2023-12-01T22:06:52.544 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:06:52.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:52.935 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:55.949 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:56.336 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:59.350 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:06:59.739 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:06:59.752 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:06:59.753 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-12-01T22:06:59.753 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-12-01T22:06:59.892 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "leader", 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 70, 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:06:59.893 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:59.894 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:59.895 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.896 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:06:59.897 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:06:59.898 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:59.899 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.900 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.901 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:59.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:06:59.903 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.904 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:06:59.905 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.906 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:06:59.907 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:59.908 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:59.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:06:59.910 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:06:59.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:06:59.911 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:06:59.911 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:06:59.912 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:06:59.912 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-12-01T22:06:59.912 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-12-01T22:07:00.051 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 70, 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:07:00.052 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:07:00.053 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:07:00.054 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.055 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:07:00.056 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:07:00.057 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.058 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:07:00.060 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:07:00.061 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:07:00.063 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:07:00.064 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:07:00.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:07:00.066 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:07:00.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": [ 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.068 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:07:00.069 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 3 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": [ 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:07:00.070 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:07:00.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.071 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.071 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:07:00.071 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:00.072 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:07:00.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-12-01T22:07:00.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-12-01T22:07:00.216 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "peon", 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: "election_epoch": 70, 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum": [ 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: 0, 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: 1, 2023-12-01T22:07:00.217 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_age": 3, 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "required_con": "2449958747317026820", 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "required_mon": [ 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:07:00.218 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_con": "4540138314316775423", 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "quorum_mon": [ 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:07:00.219 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: "outside_quorum": [], 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "extra_probe_peers": [], 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "sync_provider": [], 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": { 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 1, 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e", 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2023-12-01T22:00:10.779807Z", 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release": 16, 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "min_mon_release_name": "pacific", 2023-12-01T22:07:00.221 INFO:teuthology.orchestra.run.smithi039.stdout: "election_strategy": 1, 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "disallowed_leaders: ": "", 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false, 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "tiebreaker_mon": "", 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_ranks: ": "", 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "features": { 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "persistent": [ 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "kraken", 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "luminous", 2023-12-01T22:07:00.222 INFO:teuthology.orchestra.run.smithi039.stdout: "mimic", 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus", 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus", 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific", 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "elector-pinging" 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "optional": [] 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.223 INFO:teuthology.orchestra.run.smithi039.stdout: "mons": [ 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 0, 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "a", 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3300", 2023-12-01T22:07:00.224 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789", 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6789/0", 2023-12-01T22:07:00.225 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6789/0", 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 1, 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "b", 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:07:00.226 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:3300", 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789", 2023-12-01T22:07:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.170:6789/0", 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.170:6789/0", 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:07:00.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.229 INFO:teuthology.orchestra.run.smithi039.stdout: "rank": 2, 2023-12-01T22:07:00.229 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "c", 2023-12-01T22:07:00.229 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2023-12-01T22:07:00.229 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2023-12-01T22:07:00.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:3301", 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790", 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 0 2023-12-01T22:07:00.230 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6790/0", 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6790/0", 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: "priority": 0, 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "{}" 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.231 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_map": { 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": [ 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.232 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: "client": [ 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: "features": "0x3f01cfbdfffdffff", 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: "release": "luminous", 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 1 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": false 2023-12-01T22:07:00.233 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:00.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-12-01T22:07:00.235 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-12-01T22:07:00.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-12-01T22:07:00.610 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.609+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.610 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.609+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=6,logm=66} crc {auth=312348833,config=3140514009,health=481067966,kv=878499658,logm=4243488298}) 2023-12-01T22:07:00.610 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.609+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=312348833,config=3140514009,health=481067966,kv=1111241683,logm=3434982647}) 2023-12-01T22:07:00.611 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.609+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.611 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.609+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=6,logm=66} crc {auth=312348833,config=3140514009,health=481067966,kv=878499658,logm=4243488298}) 2023-12-01T22:07:00.611 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.609+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=6,logm=68} crc {auth=2106450303,config=3140514009,health=681234530,kv=874871471,logm=991395572}) 2023-12-01T22:07:00.612 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.610+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.612 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.610+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642018387}) 2023-12-01T22:07:00.612 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.610+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1147505065}) 2023-12-01T22:07:00.612 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.610+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.612 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.610+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642018387}) 2023-12-01T22:07:00.612 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.610+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903825010}) 2023-12-01T22:07:00.614 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.612+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.614 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.612+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701165892}) 2023-12-01T22:07:00.614 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.613+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587779391}) 2023-12-01T22:07:00.614 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.613+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.614 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.613+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701165892}) 2023-12-01T22:07:00.614 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.613+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2863975588}) 2023-12-01T22:07:00.617 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.616+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.617 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.616+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053999505}) 2023-12-01T22:07:00.617 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.616+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1220062333}) 2023-12-01T22:07:00.617 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.616+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.617 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.616+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053999505}) 2023-12-01T22:07:00.617 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.616+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180426001}) 2023-12-01T22:07:00.618 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.617+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.619 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.617+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=3,osd_pg_creating=1,osdmap=45} crc {logm=480403157,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1286984087,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2834676970,osd_pg_creating=2467789658,osdmap=2413602667}) 2023-12-01T22:07:00.619 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.617+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=22,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=11,mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=44} crc {logm=1343310207,mdsmap=3585761405,mgr=1224443669,mgr_command_descs=1448462394,mgrstat=4284369924,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=2467789658,osdmap=1491499231}) 2023-12-01T22:07:00.619 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.617+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.619 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.617+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3,monmap=3,osd_metadata=3,osd_pg_creating=1,osdmap=45} crc {logm=480403157,mdsmap=1531151339,mgr=1224443669,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=1286984087,mon_config_key=2250838263,monmap=3233185808,osd_metadata=2834676970,osd_pg_creating=2467789658,osdmap=2413602667}) 2023-12-01T22:07:00.619 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.617+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=42} crc {logm=3525539807,mdsmap=1531151339,mgr=1268538720,mgr_command_descs=1448462394,mgr_metadata=1180883800,mgrstat=2987898590,mon_config_key=3439908406,monmap=3233185808,osd_metadata=2309343068,osd_pg_creating=2467789658,osdmap=4260389295}) 2023-12-01T22:07:00.620 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.618+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.620 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.618+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2081684467}) 2023-12-01T22:07:00.620 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.618+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=617862164}) 2023-12-01T22:07:00.620 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.618+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.620 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.618+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2081684467}) 2023-12-01T22:07:00.621 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.618+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=964533693}) 2023-12-01T22:07:00.626 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.624+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.626 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.624+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=976471626}) 2023-12-01T22:07:00.626 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.624+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=46} crc {osdmap=2952489684}) 2023-12-01T22:07:00.626 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.625+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-12-01T22:07:00.626 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.625+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=976471626}) 2023-12-01T22:07:00.626 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-12-01T22:07:00.625+0000 7fa1fa16b700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=2566075175}) 2023-12-01T22:07:00.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2023-12-01T22:07:01.629 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-12-01T22:07:01.629 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-12-01T22:07:02.017 INFO:teuthology.orchestra.run.smithi039.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138314316775423","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","modified":"2023-12-01T22:00:10.779807Z","created":"2023-12-01T22:00:10.779807Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3301","nonce":0},{"type":"v1","addr":"172.21.15.39:6790","nonce":0}]},"addr":"172.21.15.39:6790/0","public_addr":"172.21.15.39:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-12-01T22:07:02.031 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-12-01T22:07:02.031 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-01T22:07:02.041 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-12-01T22:07:02.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T22:07:02.344 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:02.344 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":135,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:06:14.941418+0000","last_up_change":"2023-12-01T22:00:51.347488+0000","last_in_change":"2023-12-01T22:05:53.343041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":65,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-01T22:00:51.401944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T22:00:53.575133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2023-12-01T22:06:09.617480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2023-12-01T22:06:11.258161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":86687},{"type":"v1","addr":"172.21.15.39:6815","nonce":86687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6817","nonce":86687},{"type":"v1","addr":"172.21.15.39:6819","nonce":86687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":86687},{"type":"v1","addr":"172.21.15.39:6823","nonce":86687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":86687},{"type":"v1","addr":"172.21.15.39:6821","nonce":86687}]},"public_addr":"172.21.15.39:6815/86687","cluster_addr":"172.21.15.39:6819/86687","heartbeat_back_addr":"172.21.15.39:6823/86687","heartbeat_front_addr":"172.21.15.39:6821/86687","state":["exists","up"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":86688},{"type":"v1","addr":"172.21.15.39:6809","nonce":86688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":86688},{"type":"v1","addr":"172.21.15.39:6811","nonce":86688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":86688},{"type":"v1","addr":"172.21.15.39:6818","nonce":86688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":86688},{"type":"v1","addr":"172.21.15.39:6814","nonce":86688}]},"public_addr":"172.21.15.39:6809/86688","cluster_addr":"172.21.15.39:6811/86688","heartbeat_back_addr":"172.21.15.39:6818/86688","heartbeat_front_addr":"172.21.15.39:6814/86688","state":["exists","up"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":86689},{"type":"v1","addr":"172.21.15.39:6801","nonce":86689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":86689},{"type":"v1","addr":"172.21.15.39:6803","nonce":86689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":86689},{"type":"v1","addr":"172.21.15.39:6807","nonce":86689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":86689},{"type":"v1","addr":"172.21.15.39:6805","nonce":86689}]},"public_addr":"172.21.15.39:6801/86689","cluster_addr":"172.21.15.39:6803/86689","heartbeat_back_addr":"172.21.15.39:6807/86689","heartbeat_front_addr":"172.21.15.39:6805/86689","state":["exists","up"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86579},{"type":"v1","addr":"172.21.15.170:6801","nonce":86579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86579},{"type":"v1","addr":"172.21.15.170:6803","nonce":86579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86579},{"type":"v1","addr":"172.21.15.170:6807","nonce":86579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86579},{"type":"v1","addr":"172.21.15.170:6805","nonce":86579}]},"public_addr":"172.21.15.170:6801/86579","cluster_addr":"172.21.15.170:6803/86579","heartbeat_back_addr":"172.21.15.170:6807/86579","heartbeat_front_addr":"172.21.15.170:6805/86579","state":["exists","up"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86580},{"type":"v1","addr":"172.21.15.170:6809","nonce":86580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86580},{"type":"v1","addr":"172.21.15.170:6811","nonce":86580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86580},{"type":"v1","addr":"172.21.15.170:6815","nonce":86580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86580},{"type":"v1","addr":"172.21.15.170:6813","nonce":86580}]},"public_addr":"172.21.15.170:6809/86580","cluster_addr":"172.21.15.170:6811/86580","heartbeat_back_addr":"172.21.15.170:6815/86580","heartbeat_front_addr":"172.21.15.170:6813/86580","state":["exists","up"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":86581},{"type":"v1","addr":"172.21.15.170:6817","nonce":86581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86581},{"type":"v1","addr":"172.21.15.170:6819","nonce":86581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86581},{"type":"v1","addr":"172.21.15.170:6823","nonce":86581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86581},{"type":"v1","addr":"172.21.15.170:6821","nonce":86581}]},"public_addr":"172.21.15.170:6817/86581","cluster_addr":"172.21.15.170:6819/86581","heartbeat_back_addr":"172.21.15.170:6823/86581","heartbeat_front_addr":"172.21.15.170:6821/86581","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.528486+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.550125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.433234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.495896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.515277+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.545278+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:07:02.358 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-01T22:07:02.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:07:02.664 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:07:02.676 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-12-01T22:07:01.370409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1791480,"kb_used_data":8904,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564439560,"statfs":{"total":579820584960,"available":577986109440,"internally_reserved":0,"allocated":9117696,"data_stored":4941552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629798+0000","last_change":"2023-12-01T22:06:13.501347+0000","last_active":"2023-12-01T22:06:14.629798+0000","last_peered":"2023-12-01T22:06:14.629798+0000","last_clean":"2023-12-01T22:06:14.629798+0000","last_became_active":"2023-12-01T22:06:13.501014+0000","last_became_peered":"2023-12-01T22:06:13.501014+0000","last_unstale":"2023-12-01T22:06:14.629798+0000","last_undegraded":"2023-12-01T22:06:14.629798+0000","last_fullsized":"2023-12-01T22:06:14.629798+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502631+0000","last_change":"2023-12-01T22:06:11.483134+0000","last_active":"2023-12-01T22:06:14.502631+0000","last_peered":"2023-12-01T22:06:14.502631+0000","last_clean":"2023-12-01T22:06:14.502631+0000","last_became_active":"2023-12-01T22:06:11.482843+0000","last_became_peered":"2023-12-01T22:06:11.482843+0000","last_unstale":"2023-12-01T22:06:14.502631+0000","last_undegraded":"2023-12-01T22:06:14.502631+0000","last_fullsized":"2023-12-01T22:06:14.502631+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629484+0000","last_change":"2023-12-01T22:06:13.501342+0000","last_active":"2023-12-01T22:06:14.629484+0000","last_peered":"2023-12-01T22:06:14.629484+0000","last_clean":"2023-12-01T22:06:14.629484+0000","last_became_active":"2023-12-01T22:06:13.500964+0000","last_became_peered":"2023-12-01T22:06:13.500964+0000","last_unstale":"2023-12-01T22:06:14.629484+0000","last_undegraded":"2023-12-01T22:06:14.629484+0000","last_fullsized":"2023-12-01T22:06:14.629484+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502610+0000","last_change":"2023-12-01T22:06:11.484990+0000","last_active":"2023-12-01T22:06:14.502610+0000","last_peered":"2023-12-01T22:06:14.502610+0000","last_clean":"2023-12-01T22:06:14.502610+0000","last_became_active":"2023-12-01T22:06:11.484646+0000","last_became_peered":"2023-12-01T22:06:11.484646+0000","last_unstale":"2023-12-01T22:06:14.502610+0000","last_undegraded":"2023-12-01T22:06:14.502610+0000","last_fullsized":"2023-12-01T22:06:14.502610+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502563+0000","last_change":"2023-12-01T22:06:13.503473+0000","last_active":"2023-12-01T22:06:14.502563+0000","last_peered":"2023-12-01T22:06:14.502563+0000","last_clean":"2023-12-01T22:06:14.502563+0000","last_became_active":"2023-12-01T22:06:13.503126+0000","last_became_peered":"2023-12-01T22:06:13.503126+0000","last_unstale":"2023-12-01T22:06:14.502563+0000","last_undegraded":"2023-12-01T22:06:14.502563+0000","last_fullsized":"2023-12-01T22:06:14.502563+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629795+0000","last_change":"2023-12-01T22:06:11.483341+0000","last_active":"2023-12-01T22:06:14.629795+0000","last_peered":"2023-12-01T22:06:14.629795+0000","last_clean":"2023-12-01T22:06:14.629795+0000","last_became_active":"2023-12-01T22:06:11.482990+0000","last_became_peered":"2023-12-01T22:06:11.482990+0000","last_unstale":"2023-12-01T22:06:14.629795+0000","last_undegraded":"2023-12-01T22:06:14.629795+0000","last_fullsized":"2023-12-01T22:06:14.629795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502999+0000","last_change":"2023-12-01T22:06:13.501802+0000","last_active":"2023-12-01T22:06:14.502999+0000","last_peered":"2023-12-01T22:06:14.502999+0000","last_clean":"2023-12-01T22:06:14.502999+0000","last_became_active":"2023-12-01T22:06:13.501277+0000","last_became_peered":"2023-12-01T22:06:13.501277+0000","last_unstale":"2023-12-01T22:06:14.502999+0000","last_undegraded":"2023-12-01T22:06:14.502999+0000","last_fullsized":"2023-12-01T22:06:14.502999+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629580+0000","last_change":"2023-12-01T22:06:11.484579+0000","last_active":"2023-12-01T22:06:14.629580+0000","last_peered":"2023-12-01T22:06:14.629580+0000","last_clean":"2023-12-01T22:06:14.629580+0000","last_became_active":"2023-12-01T22:06:11.484255+0000","last_became_peered":"2023-12-01T22:06:11.484255+0000","last_unstale":"2023-12-01T22:06:14.629580+0000","last_undegraded":"2023-12-01T22:06:14.629580+0000","last_fullsized":"2023-12-01T22:06:14.629580+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502458+0000","last_change":"2023-12-01T22:06:13.503682+0000","last_active":"2023-12-01T22:06:14.502458+0000","last_peered":"2023-12-01T22:06:14.502458+0000","last_clean":"2023-12-01T22:06:14.502458+0000","last_became_active":"2023-12-01T22:06:13.503327+0000","last_became_peered":"2023-12-01T22:06:13.503327+0000","last_unstale":"2023-12-01T22:06:14.502458+0000","last_undegraded":"2023-12-01T22:06:14.502458+0000","last_fullsized":"2023-12-01T22:06:14.502458+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502569+0000","last_change":"2023-12-01T22:06:11.482204+0000","last_active":"2023-12-01T22:06:14.502569+0000","last_peered":"2023-12-01T22:06:14.502569+0000","last_clean":"2023-12-01T22:06:14.502569+0000","last_became_active":"2023-12-01T22:06:11.481865+0000","last_became_peered":"2023-12-01T22:06:11.481865+0000","last_unstale":"2023-12-01T22:06:14.502569+0000","last_undegraded":"2023-12-01T22:06:14.502569+0000","last_fullsized":"2023-12-01T22:06:14.502569+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.628619+0000","last_change":"2023-12-01T22:06:13.500317+0000","last_active":"2023-12-01T22:06:14.628619+0000","last_peered":"2023-12-01T22:06:14.628619+0000","last_clean":"2023-12-01T22:06:14.628619+0000","last_became_active":"2023-12-01T22:06:13.500005+0000","last_became_peered":"2023-12-01T22:06:13.500005+0000","last_unstale":"2023-12-01T22:06:14.628619+0000","last_undegraded":"2023-12-01T22:06:14.628619+0000","last_fullsized":"2023-12-01T22:06:14.628619+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629650+0000","last_change":"2023-12-01T22:06:11.484953+0000","last_active":"2023-12-01T22:06:14.629650+0000","last_peered":"2023-12-01T22:06:14.629650+0000","last_clean":"2023-12-01T22:06:14.629650+0000","last_became_active":"2023-12-01T22:06:11.484596+0000","last_became_peered":"2023-12-01T22:06:11.484596+0000","last_unstale":"2023-12-01T22:06:14.629650+0000","last_undegraded":"2023-12-01T22:06:14.629650+0000","last_fullsized":"2023-12-01T22:06:14.629650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629536+0000","last_change":"2023-12-01T22:06:13.502547+0000","last_active":"2023-12-01T22:06:14.629536+0000","last_peered":"2023-12-01T22:06:14.629536+0000","last_clean":"2023-12-01T22:06:14.629536+0000","last_became_active":"2023-12-01T22:06:13.502190+0000","last_became_peered":"2023-12-01T22:06:13.502190+0000","last_unstale":"2023-12-01T22:06:14.629536+0000","last_undegraded":"2023-12-01T22:06:14.629536+0000","last_fullsized":"2023-12-01T22:06:14.629536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629855+0000","last_change":"2023-12-01T22:06:11.484374+0000","last_active":"2023-12-01T22:06:14.629855+0000","last_peered":"2023-12-01T22:06:14.629855+0000","last_clean":"2023-12-01T22:06:14.629855+0000","last_became_active":"2023-12-01T22:06:11.484068+0000","last_became_peered":"2023-12-01T22:06:11.484068+0000","last_unstale":"2023-12-01T22:06:14.629855+0000","last_undegraded":"2023-12-01T22:06:14.629855+0000","last_fullsized":"2023-12-01T22:06:14.629855+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629214+0000","last_change":"2023-12-01T22:06:13.503127+0000","last_active":"2023-12-01T22:06:14.629214+0000","last_peered":"2023-12-01T22:06:14.629214+0000","last_clean":"2023-12-01T22:06:14.629214+0000","last_became_active":"2023-12-01T22:06:13.502721+0000","last_became_peered":"2023-12-01T22:06:13.502721+0000","last_unstale":"2023-12-01T22:06:14.629214+0000","last_undegraded":"2023-12-01T22:06:14.629214+0000","last_fullsized":"2023-12-01T22:06:14.629214+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629370+0000","last_change":"2023-12-01T22:06:11.486324+0000","last_active":"2023-12-01T22:06:14.629370+0000","last_peered":"2023-12-01T22:06:14.629370+0000","last_clean":"2023-12-01T22:06:14.629370+0000","last_became_active":"2023-12-01T22:06:11.486042+0000","last_became_peered":"2023-12-01T22:06:11.486042+0000","last_unstale":"2023-12-01T22:06:14.629370+0000","last_undegraded":"2023-12-01T22:06:14.629370+0000","last_fullsized":"2023-12-01T22:06:14.629370+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629675+0000","last_change":"2023-12-01T22:06:04.970068+0000","last_active":"2023-12-01T22:06:14.629675+0000","last_peered":"2023-12-01T22:06:14.629675+0000","last_clean":"2023-12-01T22:06:14.629675+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.629675+0000","last_undegraded":"2023-12-01T22:06:14.629675+0000","last_fullsized":"2023-12-01T22:06:14.629675+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":153,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502444+0000","last_change":"2023-12-01T22:06:04.826303+0000","last_active":"2023-12-01T22:06:14.502444+0000","last_peered":"2023-12-01T22:06:14.502444+0000","last_clean":"2023-12-01T22:06:14.502444+0000","last_became_active":"2023-12-01T22:06:04.826034+0000","last_became_peered":"2023-12-01T22:06:04.826034+0000","last_unstale":"2023-12-01T22:06:14.502444+0000","last_undegraded":"2023-12-01T22:06:14.502444+0000","last_fullsized":"2023-12-01T22:06:14.502444+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629272+0000","last_change":"2023-12-01T22:06:04.826573+0000","last_active":"2023-12-01T22:06:14.629272+0000","last_peered":"2023-12-01T22:06:14.629272+0000","last_clean":"2023-12-01T22:06:14.629272+0000","last_became_active":"2023-12-01T22:06:04.826155+0000","last_became_peered":"2023-12-01T22:06:04.826155+0000","last_unstale":"2023-12-01T22:06:14.629272+0000","last_undegraded":"2023-12-01T22:06:14.629272+0000","last_fullsized":"2023-12-01T22:06:14.629272+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":151,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502541+0000","last_change":"2023-12-01T22:06:04.825578+0000","last_active":"2023-12-01T22:06:14.502541+0000","last_peered":"2023-12-01T22:06:14.502541+0000","last_clean":"2023-12-01T22:06:14.502541+0000","last_became_active":"2023-12-01T22:06:04.825295+0000","last_became_peered":"2023-12-01T22:06:04.825295+0000","last_unstale":"2023-12-01T22:06:14.502541+0000","last_undegraded":"2023-12-01T22:06:14.502541+0000","last_fullsized":"2023-12-01T22:06:14.502541+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502665+0000","last_change":"2023-12-01T22:06:04.969610+0000","last_active":"2023-12-01T22:06:14.502665+0000","last_peered":"2023-12-01T22:06:14.502665+0000","last_clean":"2023-12-01T22:06:14.502665+0000","last_became_active":"2023-12-01T22:06:04.969355+0000","last_became_peered":"2023-12-01T22:06:04.969355+0000","last_unstale":"2023-12-01T22:06:14.502665+0000","last_undegraded":"2023-12-01T22:06:14.502665+0000","last_fullsized":"2023-12-01T22:06:14.502665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502576+0000","last_change":"2023-12-01T22:06:04.825803+0000","last_active":"2023-12-01T22:06:14.502576+0000","last_peered":"2023-12-01T22:06:14.502576+0000","last_clean":"2023-12-01T22:06:14.502576+0000","last_became_active":"2023-12-01T22:06:04.825428+0000","last_became_peered":"2023-12-01T22:06:04.825428+0000","last_unstale":"2023-12-01T22:06:14.502576+0000","last_undegraded":"2023-12-01T22:06:14.502576+0000","last_fullsized":"2023-12-01T22:06:14.502576+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502645+0000","last_change":"2023-12-01T22:06:04.969968+0000","last_active":"2023-12-01T22:06:14.502645+0000","last_peered":"2023-12-01T22:06:14.502645+0000","last_clean":"2023-12-01T22:06:14.502645+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.502645+0000","last_undegraded":"2023-12-01T22:06:14.502645+0000","last_fullsized":"2023-12-01T22:06:14.502645+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629364+0000","last_change":"2023-12-01T22:06:04.826087+0000","last_active":"2023-12-01T22:06:14.629364+0000","last_peered":"2023-12-01T22:06:14.629364+0000","last_clean":"2023-12-01T22:06:14.629364+0000","last_became_active":"2023-12-01T22:06:04.825776+0000","last_became_peered":"2023-12-01T22:06:04.825776+0000","last_unstale":"2023-12-01T22:06:14.629364+0000","last_undegraded":"2023-12-01T22:06:14.629364+0000","last_fullsized":"2023-12-01T22:06:14.629364+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.503075+0000","last_change":"2023-12-01T22:06:04.824717+0000","last_active":"2023-12-01T22:06:14.503075+0000","last_peered":"2023-12-01T22:06:14.503075+0000","last_clean":"2023-12-01T22:06:14.503075+0000","last_became_active":"2023-12-01T22:06:04.824420+0000","last_became_peered":"2023-12-01T22:06:04.824420+0000","last_unstale":"2023-12-01T22:06:14.503075+0000","last_undegraded":"2023-12-01T22:06:14.503075+0000","last_fullsized":"2023-12-01T22:06:14.503075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48099999999999998,"15min":0.49099999999999999},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.68899999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48299999999999998,"15min":0.497},"min":{"1min":0.316,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.441},{"interface":"front","average":{"1min":0.499,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71099999999999997,"5min":0.872,"15min":0.872},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33400000000000002,"15min":0.34699999999999998},"min":{"1min":0.23599999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56299999999999994,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.36699999999999999},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59599999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51500000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.379,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.496,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.38800000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640332,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.746,"15min":0.746},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.376}]},{"osd":1,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.59599999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.438},{"interface":"front","average":{"1min":0.497,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48399999999999999,"15min":0.495},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59899999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.499,"15min":0.47699999999999998},"min":{"1min":0.38400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.45900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.33500000000000002},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46800000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.245,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.501,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47899999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640333,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.496,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.61499999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.877,"15min":0.877},"last":0.46500000000000002}]},{"osd":1,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67200000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80800000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.436}]},{"osd":2,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.59099999999999997,"15min":0.53200000000000003},"min":{"1min":0.28799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69199999999999995,"5min":8.4689999999999994,"15min":8.4689999999999994},"last":0.497},{"interface":"front","average":{"1min":0.504,"5min":0.60699999999999998,"15min":0.54600000000000004},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68300000000000005,"5min":7.819,"15min":7.819},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33600000000000002,"15min":0.35999999999999999},"min":{"1min":0.22700000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.45800000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.74399999999999999,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":0.317}]},{"osd":5,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.42499999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.42499999999999999,"5min":3.468,"15min":3.468},"last":0.38300000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.45800000000000002,"15min":0.374},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.53200000000000003,"5min":12.333,"15min":12.333},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.39100000000000001,"15min":0.36799999999999999},"min":{"1min":0.27400000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":4.0609999999999999,"15min":4.0609999999999999},"last":0.313}]},{"osd":1,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"min":{"1min":0.30399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51400000000000001,"5min":12.217000000000001,"15min":12.217000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.56799999999999995,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":13.824,"15min":13.824},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.82999999999999996,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.69299999999999995,"5min":29.079999999999998,"15min":29.079999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.70099999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76800000000000002,"5min":16.920999999999999,"15min":16.920999999999999},"last":0.504}]},{"osd":4,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67400000000000004,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.52700000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.53400000000000003},"min":{"1min":0.41899999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.72499999999999998,"5min":4.0149999999999997,"15min":4.0149999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72699999999999998,"5min":2.524,"15min":2.524},"last":0.60099999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640332,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.48199999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36399999999999999,"15min":0.373},"min":{"1min":0.24199999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.27600000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.57199999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38600000000000001,"15min":0.38},"min":{"1min":0.29999999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56000000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.316}]},{"osd":3,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76600000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69499999999999995,"5min":0.751,"15min":0.751},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.50900000000000001},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.64800000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.748,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45600000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.41699999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76300000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.70999999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.40500000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33900000000000002,"15min":0.34200000000000003},"min":{"1min":0.23300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.51300000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.371,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58799999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.41399999999999998},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67200000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27400000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79000000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.436}]},{"osd":5,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:07:02.679 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:07:02.986 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:02.988 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:07:02.999 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-12-01T22:07:01.370409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1791480,"kb_used_data":8904,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564439560,"statfs":{"total":579820584960,"available":577986109440,"internally_reserved":0,"allocated":9117696,"data_stored":4941552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629798+0000","last_change":"2023-12-01T22:06:13.501347+0000","last_active":"2023-12-01T22:06:14.629798+0000","last_peered":"2023-12-01T22:06:14.629798+0000","last_clean":"2023-12-01T22:06:14.629798+0000","last_became_active":"2023-12-01T22:06:13.501014+0000","last_became_peered":"2023-12-01T22:06:13.501014+0000","last_unstale":"2023-12-01T22:06:14.629798+0000","last_undegraded":"2023-12-01T22:06:14.629798+0000","last_fullsized":"2023-12-01T22:06:14.629798+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502631+0000","last_change":"2023-12-01T22:06:11.483134+0000","last_active":"2023-12-01T22:06:14.502631+0000","last_peered":"2023-12-01T22:06:14.502631+0000","last_clean":"2023-12-01T22:06:14.502631+0000","last_became_active":"2023-12-01T22:06:11.482843+0000","last_became_peered":"2023-12-01T22:06:11.482843+0000","last_unstale":"2023-12-01T22:06:14.502631+0000","last_undegraded":"2023-12-01T22:06:14.502631+0000","last_fullsized":"2023-12-01T22:06:14.502631+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629484+0000","last_change":"2023-12-01T22:06:13.501342+0000","last_active":"2023-12-01T22:06:14.629484+0000","last_peered":"2023-12-01T22:06:14.629484+0000","last_clean":"2023-12-01T22:06:14.629484+0000","last_became_active":"2023-12-01T22:06:13.500964+0000","last_became_peered":"2023-12-01T22:06:13.500964+0000","last_unstale":"2023-12-01T22:06:14.629484+0000","last_undegraded":"2023-12-01T22:06:14.629484+0000","last_fullsized":"2023-12-01T22:06:14.629484+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502610+0000","last_change":"2023-12-01T22:06:11.484990+0000","last_active":"2023-12-01T22:06:14.502610+0000","last_peered":"2023-12-01T22:06:14.502610+0000","last_clean":"2023-12-01T22:06:14.502610+0000","last_became_active":"2023-12-01T22:06:11.484646+0000","last_became_peered":"2023-12-01T22:06:11.484646+0000","last_unstale":"2023-12-01T22:06:14.502610+0000","last_undegraded":"2023-12-01T22:06:14.502610+0000","last_fullsized":"2023-12-01T22:06:14.502610+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502563+0000","last_change":"2023-12-01T22:06:13.503473+0000","last_active":"2023-12-01T22:06:14.502563+0000","last_peered":"2023-12-01T22:06:14.502563+0000","last_clean":"2023-12-01T22:06:14.502563+0000","last_became_active":"2023-12-01T22:06:13.503126+0000","last_became_peered":"2023-12-01T22:06:13.503126+0000","last_unstale":"2023-12-01T22:06:14.502563+0000","last_undegraded":"2023-12-01T22:06:14.502563+0000","last_fullsized":"2023-12-01T22:06:14.502563+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629795+0000","last_change":"2023-12-01T22:06:11.483341+0000","last_active":"2023-12-01T22:06:14.629795+0000","last_peered":"2023-12-01T22:06:14.629795+0000","last_clean":"2023-12-01T22:06:14.629795+0000","last_became_active":"2023-12-01T22:06:11.482990+0000","last_became_peered":"2023-12-01T22:06:11.482990+0000","last_unstale":"2023-12-01T22:06:14.629795+0000","last_undegraded":"2023-12-01T22:06:14.629795+0000","last_fullsized":"2023-12-01T22:06:14.629795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502999+0000","last_change":"2023-12-01T22:06:13.501802+0000","last_active":"2023-12-01T22:06:14.502999+0000","last_peered":"2023-12-01T22:06:14.502999+0000","last_clean":"2023-12-01T22:06:14.502999+0000","last_became_active":"2023-12-01T22:06:13.501277+0000","last_became_peered":"2023-12-01T22:06:13.501277+0000","last_unstale":"2023-12-01T22:06:14.502999+0000","last_undegraded":"2023-12-01T22:06:14.502999+0000","last_fullsized":"2023-12-01T22:06:14.502999+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629580+0000","last_change":"2023-12-01T22:06:11.484579+0000","last_active":"2023-12-01T22:06:14.629580+0000","last_peered":"2023-12-01T22:06:14.629580+0000","last_clean":"2023-12-01T22:06:14.629580+0000","last_became_active":"2023-12-01T22:06:11.484255+0000","last_became_peered":"2023-12-01T22:06:11.484255+0000","last_unstale":"2023-12-01T22:06:14.629580+0000","last_undegraded":"2023-12-01T22:06:14.629580+0000","last_fullsized":"2023-12-01T22:06:14.629580+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502458+0000","last_change":"2023-12-01T22:06:13.503682+0000","last_active":"2023-12-01T22:06:14.502458+0000","last_peered":"2023-12-01T22:06:14.502458+0000","last_clean":"2023-12-01T22:06:14.502458+0000","last_became_active":"2023-12-01T22:06:13.503327+0000","last_became_peered":"2023-12-01T22:06:13.503327+0000","last_unstale":"2023-12-01T22:06:14.502458+0000","last_undegraded":"2023-12-01T22:06:14.502458+0000","last_fullsized":"2023-12-01T22:06:14.502458+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502569+0000","last_change":"2023-12-01T22:06:11.482204+0000","last_active":"2023-12-01T22:06:14.502569+0000","last_peered":"2023-12-01T22:06:14.502569+0000","last_clean":"2023-12-01T22:06:14.502569+0000","last_became_active":"2023-12-01T22:06:11.481865+0000","last_became_peered":"2023-12-01T22:06:11.481865+0000","last_unstale":"2023-12-01T22:06:14.502569+0000","last_undegraded":"2023-12-01T22:06:14.502569+0000","last_fullsized":"2023-12-01T22:06:14.502569+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.628619+0000","last_change":"2023-12-01T22:06:13.500317+0000","last_active":"2023-12-01T22:06:14.628619+0000","last_peered":"2023-12-01T22:06:14.628619+0000","last_clean":"2023-12-01T22:06:14.628619+0000","last_became_active":"2023-12-01T22:06:13.500005+0000","last_became_peered":"2023-12-01T22:06:13.500005+0000","last_unstale":"2023-12-01T22:06:14.628619+0000","last_undegraded":"2023-12-01T22:06:14.628619+0000","last_fullsized":"2023-12-01T22:06:14.628619+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629650+0000","last_change":"2023-12-01T22:06:11.484953+0000","last_active":"2023-12-01T22:06:14.629650+0000","last_peered":"2023-12-01T22:06:14.629650+0000","last_clean":"2023-12-01T22:06:14.629650+0000","last_became_active":"2023-12-01T22:06:11.484596+0000","last_became_peered":"2023-12-01T22:06:11.484596+0000","last_unstale":"2023-12-01T22:06:14.629650+0000","last_undegraded":"2023-12-01T22:06:14.629650+0000","last_fullsized":"2023-12-01T22:06:14.629650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629536+0000","last_change":"2023-12-01T22:06:13.502547+0000","last_active":"2023-12-01T22:06:14.629536+0000","last_peered":"2023-12-01T22:06:14.629536+0000","last_clean":"2023-12-01T22:06:14.629536+0000","last_became_active":"2023-12-01T22:06:13.502190+0000","last_became_peered":"2023-12-01T22:06:13.502190+0000","last_unstale":"2023-12-01T22:06:14.629536+0000","last_undegraded":"2023-12-01T22:06:14.629536+0000","last_fullsized":"2023-12-01T22:06:14.629536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629855+0000","last_change":"2023-12-01T22:06:11.484374+0000","last_active":"2023-12-01T22:06:14.629855+0000","last_peered":"2023-12-01T22:06:14.629855+0000","last_clean":"2023-12-01T22:06:14.629855+0000","last_became_active":"2023-12-01T22:06:11.484068+0000","last_became_peered":"2023-12-01T22:06:11.484068+0000","last_unstale":"2023-12-01T22:06:14.629855+0000","last_undegraded":"2023-12-01T22:06:14.629855+0000","last_fullsized":"2023-12-01T22:06:14.629855+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629214+0000","last_change":"2023-12-01T22:06:13.503127+0000","last_active":"2023-12-01T22:06:14.629214+0000","last_peered":"2023-12-01T22:06:14.629214+0000","last_clean":"2023-12-01T22:06:14.629214+0000","last_became_active":"2023-12-01T22:06:13.502721+0000","last_became_peered":"2023-12-01T22:06:13.502721+0000","last_unstale":"2023-12-01T22:06:14.629214+0000","last_undegraded":"2023-12-01T22:06:14.629214+0000","last_fullsized":"2023-12-01T22:06:14.629214+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629370+0000","last_change":"2023-12-01T22:06:11.486324+0000","last_active":"2023-12-01T22:06:14.629370+0000","last_peered":"2023-12-01T22:06:14.629370+0000","last_clean":"2023-12-01T22:06:14.629370+0000","last_became_active":"2023-12-01T22:06:11.486042+0000","last_became_peered":"2023-12-01T22:06:11.486042+0000","last_unstale":"2023-12-01T22:06:14.629370+0000","last_undegraded":"2023-12-01T22:06:14.629370+0000","last_fullsized":"2023-12-01T22:06:14.629370+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629675+0000","last_change":"2023-12-01T22:06:04.970068+0000","last_active":"2023-12-01T22:06:14.629675+0000","last_peered":"2023-12-01T22:06:14.629675+0000","last_clean":"2023-12-01T22:06:14.629675+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.629675+0000","last_undegraded":"2023-12-01T22:06:14.629675+0000","last_fullsized":"2023-12-01T22:06:14.629675+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":153,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502444+0000","last_change":"2023-12-01T22:06:04.826303+0000","last_active":"2023-12-01T22:06:14.502444+0000","last_peered":"2023-12-01T22:06:14.502444+0000","last_clean":"2023-12-01T22:06:14.502444+0000","last_became_active":"2023-12-01T22:06:04.826034+0000","last_became_peered":"2023-12-01T22:06:04.826034+0000","last_unstale":"2023-12-01T22:06:14.502444+0000","last_undegraded":"2023-12-01T22:06:14.502444+0000","last_fullsized":"2023-12-01T22:06:14.502444+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629272+0000","last_change":"2023-12-01T22:06:04.826573+0000","last_active":"2023-12-01T22:06:14.629272+0000","last_peered":"2023-12-01T22:06:14.629272+0000","last_clean":"2023-12-01T22:06:14.629272+0000","last_became_active":"2023-12-01T22:06:04.826155+0000","last_became_peered":"2023-12-01T22:06:04.826155+0000","last_unstale":"2023-12-01T22:06:14.629272+0000","last_undegraded":"2023-12-01T22:06:14.629272+0000","last_fullsized":"2023-12-01T22:06:14.629272+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":151,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502541+0000","last_change":"2023-12-01T22:06:04.825578+0000","last_active":"2023-12-01T22:06:14.502541+0000","last_peered":"2023-12-01T22:06:14.502541+0000","last_clean":"2023-12-01T22:06:14.502541+0000","last_became_active":"2023-12-01T22:06:04.825295+0000","last_became_peered":"2023-12-01T22:06:04.825295+0000","last_unstale":"2023-12-01T22:06:14.502541+0000","last_undegraded":"2023-12-01T22:06:14.502541+0000","last_fullsized":"2023-12-01T22:06:14.502541+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502665+0000","last_change":"2023-12-01T22:06:04.969610+0000","last_active":"2023-12-01T22:06:14.502665+0000","last_peered":"2023-12-01T22:06:14.502665+0000","last_clean":"2023-12-01T22:06:14.502665+0000","last_became_active":"2023-12-01T22:06:04.969355+0000","last_became_peered":"2023-12-01T22:06:04.969355+0000","last_unstale":"2023-12-01T22:06:14.502665+0000","last_undegraded":"2023-12-01T22:06:14.502665+0000","last_fullsized":"2023-12-01T22:06:14.502665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502576+0000","last_change":"2023-12-01T22:06:04.825803+0000","last_active":"2023-12-01T22:06:14.502576+0000","last_peered":"2023-12-01T22:06:14.502576+0000","last_clean":"2023-12-01T22:06:14.502576+0000","last_became_active":"2023-12-01T22:06:04.825428+0000","last_became_peered":"2023-12-01T22:06:04.825428+0000","last_unstale":"2023-12-01T22:06:14.502576+0000","last_undegraded":"2023-12-01T22:06:14.502576+0000","last_fullsized":"2023-12-01T22:06:14.502576+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502645+0000","last_change":"2023-12-01T22:06:04.969968+0000","last_active":"2023-12-01T22:06:14.502645+0000","last_peered":"2023-12-01T22:06:14.502645+0000","last_clean":"2023-12-01T22:06:14.502645+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.502645+0000","last_undegraded":"2023-12-01T22:06:14.502645+0000","last_fullsized":"2023-12-01T22:06:14.502645+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629364+0000","last_change":"2023-12-01T22:06:04.826087+0000","last_active":"2023-12-01T22:06:14.629364+0000","last_peered":"2023-12-01T22:06:14.629364+0000","last_clean":"2023-12-01T22:06:14.629364+0000","last_became_active":"2023-12-01T22:06:04.825776+0000","last_became_peered":"2023-12-01T22:06:04.825776+0000","last_unstale":"2023-12-01T22:06:14.629364+0000","last_undegraded":"2023-12-01T22:06:14.629364+0000","last_fullsized":"2023-12-01T22:06:14.629364+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.503075+0000","last_change":"2023-12-01T22:06:04.824717+0000","last_active":"2023-12-01T22:06:14.503075+0000","last_peered":"2023-12-01T22:06:14.503075+0000","last_clean":"2023-12-01T22:06:14.503075+0000","last_became_active":"2023-12-01T22:06:04.824420+0000","last_became_peered":"2023-12-01T22:06:04.824420+0000","last_unstale":"2023-12-01T22:06:14.503075+0000","last_undegraded":"2023-12-01T22:06:14.503075+0000","last_fullsized":"2023-12-01T22:06:14.503075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48099999999999998,"15min":0.49099999999999999},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.68899999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48299999999999998,"15min":0.497},"min":{"1min":0.316,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.441},{"interface":"front","average":{"1min":0.499,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71099999999999997,"5min":0.872,"15min":0.872},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33400000000000002,"15min":0.34699999999999998},"min":{"1min":0.23599999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56299999999999994,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.36699999999999999},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59599999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51500000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.379,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.496,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.38800000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640332,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.746,"15min":0.746},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.376}]},{"osd":1,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.59599999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.438},{"interface":"front","average":{"1min":0.497,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48399999999999999,"15min":0.495},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59899999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.499,"15min":0.47699999999999998},"min":{"1min":0.38400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.45900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.33500000000000002},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46800000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.245,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.501,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47899999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640333,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.496,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.61499999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.877,"15min":0.877},"last":0.46500000000000002}]},{"osd":1,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67200000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80800000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.436}]},{"osd":2,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.59099999999999997,"15min":0.53200000000000003},"min":{"1min":0.28799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69199999999999995,"5min":8.4689999999999994,"15min":8.4689999999999994},"last":0.497},{"interface":"front","average":{"1min":0.504,"5min":0.60699999999999998,"15min":0.54600000000000004},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68300000000000005,"5min":7.819,"15min":7.819},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33600000000000002,"15min":0.35999999999999999},"min":{"1min":0.22700000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.45800000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.74399999999999999,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":0.317}]},{"osd":5,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.42499999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.42499999999999999,"5min":3.468,"15min":3.468},"last":0.38300000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.45800000000000002,"15min":0.374},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.53200000000000003,"5min":12.333,"15min":12.333},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.39100000000000001,"15min":0.36799999999999999},"min":{"1min":0.27400000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":4.0609999999999999,"15min":4.0609999999999999},"last":0.313}]},{"osd":1,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"min":{"1min":0.30399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51400000000000001,"5min":12.217000000000001,"15min":12.217000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.56799999999999995,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":13.824,"15min":13.824},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.82999999999999996,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.69299999999999995,"5min":29.079999999999998,"15min":29.079999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.70099999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76800000000000002,"5min":16.920999999999999,"15min":16.920999999999999},"last":0.504}]},{"osd":4,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67400000000000004,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.52700000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.53400000000000003},"min":{"1min":0.41899999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.72499999999999998,"5min":4.0149999999999997,"15min":4.0149999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72699999999999998,"5min":2.524,"15min":2.524},"last":0.60099999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640332,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.48199999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36399999999999999,"15min":0.373},"min":{"1min":0.24199999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.27600000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.57199999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38600000000000001,"15min":0.38},"min":{"1min":0.29999999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56000000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.316}]},{"osd":3,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76600000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69499999999999995,"5min":0.751,"15min":0.751},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.50900000000000001},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.64800000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.748,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45600000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.41699999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76300000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.70999999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.40500000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33900000000000002,"15min":0.34200000000000003},"min":{"1min":0.23300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.51300000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.371,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58799999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.41399999999999998},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67200000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27400000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79000000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.436}]},{"osd":5,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:07:03.001 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-01T22:07:03.001 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:07:03.311 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:03.312 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:07:03.323 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-12-01T22:07:01.370409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1791480,"kb_used_data":8904,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564439560,"statfs":{"total":579820584960,"available":577986109440,"internally_reserved":0,"allocated":9117696,"data_stored":4941552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629798+0000","last_change":"2023-12-01T22:06:13.501347+0000","last_active":"2023-12-01T22:06:14.629798+0000","last_peered":"2023-12-01T22:06:14.629798+0000","last_clean":"2023-12-01T22:06:14.629798+0000","last_became_active":"2023-12-01T22:06:13.501014+0000","last_became_peered":"2023-12-01T22:06:13.501014+0000","last_unstale":"2023-12-01T22:06:14.629798+0000","last_undegraded":"2023-12-01T22:06:14.629798+0000","last_fullsized":"2023-12-01T22:06:14.629798+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502631+0000","last_change":"2023-12-01T22:06:11.483134+0000","last_active":"2023-12-01T22:06:14.502631+0000","last_peered":"2023-12-01T22:06:14.502631+0000","last_clean":"2023-12-01T22:06:14.502631+0000","last_became_active":"2023-12-01T22:06:11.482843+0000","last_became_peered":"2023-12-01T22:06:11.482843+0000","last_unstale":"2023-12-01T22:06:14.502631+0000","last_undegraded":"2023-12-01T22:06:14.502631+0000","last_fullsized":"2023-12-01T22:06:14.502631+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629484+0000","last_change":"2023-12-01T22:06:13.501342+0000","last_active":"2023-12-01T22:06:14.629484+0000","last_peered":"2023-12-01T22:06:14.629484+0000","last_clean":"2023-12-01T22:06:14.629484+0000","last_became_active":"2023-12-01T22:06:13.500964+0000","last_became_peered":"2023-12-01T22:06:13.500964+0000","last_unstale":"2023-12-01T22:06:14.629484+0000","last_undegraded":"2023-12-01T22:06:14.629484+0000","last_fullsized":"2023-12-01T22:06:14.629484+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502610+0000","last_change":"2023-12-01T22:06:11.484990+0000","last_active":"2023-12-01T22:06:14.502610+0000","last_peered":"2023-12-01T22:06:14.502610+0000","last_clean":"2023-12-01T22:06:14.502610+0000","last_became_active":"2023-12-01T22:06:11.484646+0000","last_became_peered":"2023-12-01T22:06:11.484646+0000","last_unstale":"2023-12-01T22:06:14.502610+0000","last_undegraded":"2023-12-01T22:06:14.502610+0000","last_fullsized":"2023-12-01T22:06:14.502610+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502563+0000","last_change":"2023-12-01T22:06:13.503473+0000","last_active":"2023-12-01T22:06:14.502563+0000","last_peered":"2023-12-01T22:06:14.502563+0000","last_clean":"2023-12-01T22:06:14.502563+0000","last_became_active":"2023-12-01T22:06:13.503126+0000","last_became_peered":"2023-12-01T22:06:13.503126+0000","last_unstale":"2023-12-01T22:06:14.502563+0000","last_undegraded":"2023-12-01T22:06:14.502563+0000","last_fullsized":"2023-12-01T22:06:14.502563+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629795+0000","last_change":"2023-12-01T22:06:11.483341+0000","last_active":"2023-12-01T22:06:14.629795+0000","last_peered":"2023-12-01T22:06:14.629795+0000","last_clean":"2023-12-01T22:06:14.629795+0000","last_became_active":"2023-12-01T22:06:11.482990+0000","last_became_peered":"2023-12-01T22:06:11.482990+0000","last_unstale":"2023-12-01T22:06:14.629795+0000","last_undegraded":"2023-12-01T22:06:14.629795+0000","last_fullsized":"2023-12-01T22:06:14.629795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502999+0000","last_change":"2023-12-01T22:06:13.501802+0000","last_active":"2023-12-01T22:06:14.502999+0000","last_peered":"2023-12-01T22:06:14.502999+0000","last_clean":"2023-12-01T22:06:14.502999+0000","last_became_active":"2023-12-01T22:06:13.501277+0000","last_became_peered":"2023-12-01T22:06:13.501277+0000","last_unstale":"2023-12-01T22:06:14.502999+0000","last_undegraded":"2023-12-01T22:06:14.502999+0000","last_fullsized":"2023-12-01T22:06:14.502999+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629580+0000","last_change":"2023-12-01T22:06:11.484579+0000","last_active":"2023-12-01T22:06:14.629580+0000","last_peered":"2023-12-01T22:06:14.629580+0000","last_clean":"2023-12-01T22:06:14.629580+0000","last_became_active":"2023-12-01T22:06:11.484255+0000","last_became_peered":"2023-12-01T22:06:11.484255+0000","last_unstale":"2023-12-01T22:06:14.629580+0000","last_undegraded":"2023-12-01T22:06:14.629580+0000","last_fullsized":"2023-12-01T22:06:14.629580+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502458+0000","last_change":"2023-12-01T22:06:13.503682+0000","last_active":"2023-12-01T22:06:14.502458+0000","last_peered":"2023-12-01T22:06:14.502458+0000","last_clean":"2023-12-01T22:06:14.502458+0000","last_became_active":"2023-12-01T22:06:13.503327+0000","last_became_peered":"2023-12-01T22:06:13.503327+0000","last_unstale":"2023-12-01T22:06:14.502458+0000","last_undegraded":"2023-12-01T22:06:14.502458+0000","last_fullsized":"2023-12-01T22:06:14.502458+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502569+0000","last_change":"2023-12-01T22:06:11.482204+0000","last_active":"2023-12-01T22:06:14.502569+0000","last_peered":"2023-12-01T22:06:14.502569+0000","last_clean":"2023-12-01T22:06:14.502569+0000","last_became_active":"2023-12-01T22:06:11.481865+0000","last_became_peered":"2023-12-01T22:06:11.481865+0000","last_unstale":"2023-12-01T22:06:14.502569+0000","last_undegraded":"2023-12-01T22:06:14.502569+0000","last_fullsized":"2023-12-01T22:06:14.502569+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.628619+0000","last_change":"2023-12-01T22:06:13.500317+0000","last_active":"2023-12-01T22:06:14.628619+0000","last_peered":"2023-12-01T22:06:14.628619+0000","last_clean":"2023-12-01T22:06:14.628619+0000","last_became_active":"2023-12-01T22:06:13.500005+0000","last_became_peered":"2023-12-01T22:06:13.500005+0000","last_unstale":"2023-12-01T22:06:14.628619+0000","last_undegraded":"2023-12-01T22:06:14.628619+0000","last_fullsized":"2023-12-01T22:06:14.628619+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629650+0000","last_change":"2023-12-01T22:06:11.484953+0000","last_active":"2023-12-01T22:06:14.629650+0000","last_peered":"2023-12-01T22:06:14.629650+0000","last_clean":"2023-12-01T22:06:14.629650+0000","last_became_active":"2023-12-01T22:06:11.484596+0000","last_became_peered":"2023-12-01T22:06:11.484596+0000","last_unstale":"2023-12-01T22:06:14.629650+0000","last_undegraded":"2023-12-01T22:06:14.629650+0000","last_fullsized":"2023-12-01T22:06:14.629650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629536+0000","last_change":"2023-12-01T22:06:13.502547+0000","last_active":"2023-12-01T22:06:14.629536+0000","last_peered":"2023-12-01T22:06:14.629536+0000","last_clean":"2023-12-01T22:06:14.629536+0000","last_became_active":"2023-12-01T22:06:13.502190+0000","last_became_peered":"2023-12-01T22:06:13.502190+0000","last_unstale":"2023-12-01T22:06:14.629536+0000","last_undegraded":"2023-12-01T22:06:14.629536+0000","last_fullsized":"2023-12-01T22:06:14.629536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629855+0000","last_change":"2023-12-01T22:06:11.484374+0000","last_active":"2023-12-01T22:06:14.629855+0000","last_peered":"2023-12-01T22:06:14.629855+0000","last_clean":"2023-12-01T22:06:14.629855+0000","last_became_active":"2023-12-01T22:06:11.484068+0000","last_became_peered":"2023-12-01T22:06:11.484068+0000","last_unstale":"2023-12-01T22:06:14.629855+0000","last_undegraded":"2023-12-01T22:06:14.629855+0000","last_fullsized":"2023-12-01T22:06:14.629855+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629214+0000","last_change":"2023-12-01T22:06:13.503127+0000","last_active":"2023-12-01T22:06:14.629214+0000","last_peered":"2023-12-01T22:06:14.629214+0000","last_clean":"2023-12-01T22:06:14.629214+0000","last_became_active":"2023-12-01T22:06:13.502721+0000","last_became_peered":"2023-12-01T22:06:13.502721+0000","last_unstale":"2023-12-01T22:06:14.629214+0000","last_undegraded":"2023-12-01T22:06:14.629214+0000","last_fullsized":"2023-12-01T22:06:14.629214+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629370+0000","last_change":"2023-12-01T22:06:11.486324+0000","last_active":"2023-12-01T22:06:14.629370+0000","last_peered":"2023-12-01T22:06:14.629370+0000","last_clean":"2023-12-01T22:06:14.629370+0000","last_became_active":"2023-12-01T22:06:11.486042+0000","last_became_peered":"2023-12-01T22:06:11.486042+0000","last_unstale":"2023-12-01T22:06:14.629370+0000","last_undegraded":"2023-12-01T22:06:14.629370+0000","last_fullsized":"2023-12-01T22:06:14.629370+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629675+0000","last_change":"2023-12-01T22:06:04.970068+0000","last_active":"2023-12-01T22:06:14.629675+0000","last_peered":"2023-12-01T22:06:14.629675+0000","last_clean":"2023-12-01T22:06:14.629675+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.629675+0000","last_undegraded":"2023-12-01T22:06:14.629675+0000","last_fullsized":"2023-12-01T22:06:14.629675+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":153,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502444+0000","last_change":"2023-12-01T22:06:04.826303+0000","last_active":"2023-12-01T22:06:14.502444+0000","last_peered":"2023-12-01T22:06:14.502444+0000","last_clean":"2023-12-01T22:06:14.502444+0000","last_became_active":"2023-12-01T22:06:04.826034+0000","last_became_peered":"2023-12-01T22:06:04.826034+0000","last_unstale":"2023-12-01T22:06:14.502444+0000","last_undegraded":"2023-12-01T22:06:14.502444+0000","last_fullsized":"2023-12-01T22:06:14.502444+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629272+0000","last_change":"2023-12-01T22:06:04.826573+0000","last_active":"2023-12-01T22:06:14.629272+0000","last_peered":"2023-12-01T22:06:14.629272+0000","last_clean":"2023-12-01T22:06:14.629272+0000","last_became_active":"2023-12-01T22:06:04.826155+0000","last_became_peered":"2023-12-01T22:06:04.826155+0000","last_unstale":"2023-12-01T22:06:14.629272+0000","last_undegraded":"2023-12-01T22:06:14.629272+0000","last_fullsized":"2023-12-01T22:06:14.629272+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":151,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502541+0000","last_change":"2023-12-01T22:06:04.825578+0000","last_active":"2023-12-01T22:06:14.502541+0000","last_peered":"2023-12-01T22:06:14.502541+0000","last_clean":"2023-12-01T22:06:14.502541+0000","last_became_active":"2023-12-01T22:06:04.825295+0000","last_became_peered":"2023-12-01T22:06:04.825295+0000","last_unstale":"2023-12-01T22:06:14.502541+0000","last_undegraded":"2023-12-01T22:06:14.502541+0000","last_fullsized":"2023-12-01T22:06:14.502541+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502665+0000","last_change":"2023-12-01T22:06:04.969610+0000","last_active":"2023-12-01T22:06:14.502665+0000","last_peered":"2023-12-01T22:06:14.502665+0000","last_clean":"2023-12-01T22:06:14.502665+0000","last_became_active":"2023-12-01T22:06:04.969355+0000","last_became_peered":"2023-12-01T22:06:04.969355+0000","last_unstale":"2023-12-01T22:06:14.502665+0000","last_undegraded":"2023-12-01T22:06:14.502665+0000","last_fullsized":"2023-12-01T22:06:14.502665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502576+0000","last_change":"2023-12-01T22:06:04.825803+0000","last_active":"2023-12-01T22:06:14.502576+0000","last_peered":"2023-12-01T22:06:14.502576+0000","last_clean":"2023-12-01T22:06:14.502576+0000","last_became_active":"2023-12-01T22:06:04.825428+0000","last_became_peered":"2023-12-01T22:06:04.825428+0000","last_unstale":"2023-12-01T22:06:14.502576+0000","last_undegraded":"2023-12-01T22:06:14.502576+0000","last_fullsized":"2023-12-01T22:06:14.502576+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502645+0000","last_change":"2023-12-01T22:06:04.969968+0000","last_active":"2023-12-01T22:06:14.502645+0000","last_peered":"2023-12-01T22:06:14.502645+0000","last_clean":"2023-12-01T22:06:14.502645+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.502645+0000","last_undegraded":"2023-12-01T22:06:14.502645+0000","last_fullsized":"2023-12-01T22:06:14.502645+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629364+0000","last_change":"2023-12-01T22:06:04.826087+0000","last_active":"2023-12-01T22:06:14.629364+0000","last_peered":"2023-12-01T22:06:14.629364+0000","last_clean":"2023-12-01T22:06:14.629364+0000","last_became_active":"2023-12-01T22:06:04.825776+0000","last_became_peered":"2023-12-01T22:06:04.825776+0000","last_unstale":"2023-12-01T22:06:14.629364+0000","last_undegraded":"2023-12-01T22:06:14.629364+0000","last_fullsized":"2023-12-01T22:06:14.629364+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.503075+0000","last_change":"2023-12-01T22:06:04.824717+0000","last_active":"2023-12-01T22:06:14.503075+0000","last_peered":"2023-12-01T22:06:14.503075+0000","last_clean":"2023-12-01T22:06:14.503075+0000","last_became_active":"2023-12-01T22:06:04.824420+0000","last_became_peered":"2023-12-01T22:06:04.824420+0000","last_unstale":"2023-12-01T22:06:14.503075+0000","last_undegraded":"2023-12-01T22:06:14.503075+0000","last_fullsized":"2023-12-01T22:06:14.503075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48099999999999998,"15min":0.49099999999999999},"min":{"1min":0.25900000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.68899999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.307,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48299999999999998,"15min":0.497},"min":{"1min":0.316,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.36099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60999999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.441},{"interface":"front","average":{"1min":0.499,"5min":0.49199999999999999,"15min":0.501},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71099999999999997,"5min":0.872,"15min":0.872},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.33400000000000002,"15min":0.34699999999999998},"min":{"1min":0.23599999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56299999999999994,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.36699999999999999},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59599999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51500000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.379,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.496,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.38800000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640332,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.746,"15min":0.746},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.376}]},{"osd":1,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.59599999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.438},{"interface":"front","average":{"1min":0.497,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48399999999999999,"15min":0.495},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59899999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.499,"15min":0.47699999999999998},"min":{"1min":0.38400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.45900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.33500000000000002},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46800000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.245,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.501,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47899999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640333,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.496,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.61499999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.877,"15min":0.877},"last":0.46500000000000002}]},{"osd":1,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67200000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80800000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.436}]},{"osd":2,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.59099999999999997,"15min":0.53200000000000003},"min":{"1min":0.28799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69199999999999995,"5min":8.4689999999999994,"15min":8.4689999999999994},"last":0.497},{"interface":"front","average":{"1min":0.504,"5min":0.60699999999999998,"15min":0.54600000000000004},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68300000000000005,"5min":7.819,"15min":7.819},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33600000000000002,"15min":0.35999999999999999},"min":{"1min":0.22700000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.45800000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.74399999999999999,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":0.317}]},{"osd":5,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.42499999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.42499999999999999,"5min":3.468,"15min":3.468},"last":0.38300000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.45800000000000002,"15min":0.374},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.53200000000000003,"5min":12.333,"15min":12.333},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.39100000000000001,"15min":0.36799999999999999},"min":{"1min":0.27400000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":4.0609999999999999,"15min":4.0609999999999999},"last":0.313}]},{"osd":1,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"min":{"1min":0.30399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.51400000000000001,"5min":12.217000000000001,"15min":12.217000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.56799999999999995,"15min":0.47599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60099999999999998,"5min":13.824,"15min":13.824},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.82999999999999996,"15min":0.60599999999999998},"min":{"1min":0.36799999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.69299999999999995,"5min":29.079999999999998,"15min":29.079999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.70099999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76800000000000002,"5min":16.920999999999999,"15min":16.920999999999999},"last":0.504}]},{"osd":4,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53600000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67400000000000004,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.52700000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.53400000000000003},"min":{"1min":0.41899999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.72499999999999998,"5min":4.0149999999999997,"15min":4.0149999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72699999999999998,"5min":2.524,"15min":2.524},"last":0.60099999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640332,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.48199999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36399999999999999,"15min":0.373},"min":{"1min":0.24199999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.27600000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.57199999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38600000000000001,"15min":0.38},"min":{"1min":0.29999999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56000000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.316}]},{"osd":3,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76600000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69499999999999995,"5min":0.751,"15min":0.751},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.50900000000000001},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.64800000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.748,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45600000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.41699999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76300000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.70999999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.40500000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33900000000000002,"15min":0.34200000000000003},"min":{"1min":0.23300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.51300000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.371,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58799999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.41399999999999998},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67200000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27400000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79000000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.436}]},{"osd":5,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:07:03.325 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T22:07:03.633 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:03.633 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":135,"fsid":"ed44a2f1-e5ad-4ea4-b80a-42121a26ac1e","created":"2023-12-01T22:00:41.773050+0000","modified":"2023-12-01T22:06:14.941418+0000","last_up_change":"2023-12-01T22:00:51.347488+0000","last_in_change":"2023-12-01T22:05:53.343041+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":65,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-01T22:00:51.401944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T22:00:53.575133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2023-12-01T22:06:09.617480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2023-12-01T22:06:11.258161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"2c2da89b-1a10-4976-9a09-37fa58791d03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":86687},{"type":"v1","addr":"172.21.15.39:6815","nonce":86687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6817","nonce":86687},{"type":"v1","addr":"172.21.15.39:6819","nonce":86687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":86687},{"type":"v1","addr":"172.21.15.39:6823","nonce":86687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":86687},{"type":"v1","addr":"172.21.15.39:6821","nonce":86687}]},"public_addr":"172.21.15.39:6815/86687","cluster_addr":"172.21.15.39:6819/86687","heartbeat_back_addr":"172.21.15.39:6823/86687","heartbeat_front_addr":"172.21.15.39:6821/86687","state":["exists","up"]},{"osd":1,"uuid":"b735f840-37fe-4e12-b352-4bcdd00b86fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":86688},{"type":"v1","addr":"172.21.15.39:6809","nonce":86688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":86688},{"type":"v1","addr":"172.21.15.39:6811","nonce":86688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":86688},{"type":"v1","addr":"172.21.15.39:6818","nonce":86688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":86688},{"type":"v1","addr":"172.21.15.39:6814","nonce":86688}]},"public_addr":"172.21.15.39:6809/86688","cluster_addr":"172.21.15.39:6811/86688","heartbeat_back_addr":"172.21.15.39:6818/86688","heartbeat_front_addr":"172.21.15.39:6814/86688","state":["exists","up"]},{"osd":2,"uuid":"dfb64dc2-d0c6-446b-94e5-e2863d03584a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":86689},{"type":"v1","addr":"172.21.15.39:6801","nonce":86689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":86689},{"type":"v1","addr":"172.21.15.39:6803","nonce":86689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":86689},{"type":"v1","addr":"172.21.15.39:6807","nonce":86689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":86689},{"type":"v1","addr":"172.21.15.39:6805","nonce":86689}]},"public_addr":"172.21.15.39:6801/86689","cluster_addr":"172.21.15.39:6803/86689","heartbeat_back_addr":"172.21.15.39:6807/86689","heartbeat_front_addr":"172.21.15.39:6805/86689","state":["exists","up"]},{"osd":3,"uuid":"735a78cf-3c61-4168-b264-13fc8620d848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":86579},{"type":"v1","addr":"172.21.15.170:6801","nonce":86579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":86579},{"type":"v1","addr":"172.21.15.170:6803","nonce":86579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":86579},{"type":"v1","addr":"172.21.15.170:6807","nonce":86579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":86579},{"type":"v1","addr":"172.21.15.170:6805","nonce":86579}]},"public_addr":"172.21.15.170:6801/86579","cluster_addr":"172.21.15.170:6803/86579","heartbeat_back_addr":"172.21.15.170:6807/86579","heartbeat_front_addr":"172.21.15.170:6805/86579","state":["exists","up"]},{"osd":4,"uuid":"5d42499a-5964-4b40-8e56-00a6ac63fe90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":86580},{"type":"v1","addr":"172.21.15.170:6809","nonce":86580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":86580},{"type":"v1","addr":"172.21.15.170:6811","nonce":86580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":86580},{"type":"v1","addr":"172.21.15.170:6815","nonce":86580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":86580},{"type":"v1","addr":"172.21.15.170:6813","nonce":86580}]},"public_addr":"172.21.15.170:6809/86580","cluster_addr":"172.21.15.170:6811/86580","heartbeat_back_addr":"172.21.15.170:6815/86580","heartbeat_front_addr":"172.21.15.170:6813/86580","state":["exists","up"]},{"osd":5,"uuid":"09acc711-f491-4a2d-8976-79120d86045d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":86581},{"type":"v1","addr":"172.21.15.170:6817","nonce":86581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":86581},{"type":"v1","addr":"172.21.15.170:6819","nonce":86581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":86581},{"type":"v1","addr":"172.21.15.170:6823","nonce":86581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":86581},{"type":"v1","addr":"172.21.15.170:6821","nonce":86581}]},"public_addr":"172.21.15.170:6817/86581","cluster_addr":"172.21.15.170:6819/86581","heartbeat_back_addr":"172.21.15.170:6823/86581","heartbeat_front_addr":"172.21.15.170:6821/86581","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.528486+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.550125+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.433234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.495896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.515277+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T22:00:49.545278+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T22:07:04.647 INFO:tasks.ceph:Scrubbing osd.0 2023-12-01T22:07:04.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-01T22:07:04.803 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:04.804 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-01T22:07:04.804 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:04.815 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-01T22:07:05.129 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 0 to deep-scrub 2023-12-01T22:07:05.140 INFO:tasks.ceph:Scrubbing osd.1 2023-12-01T22:07:05.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-01T22:07:05.290 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:05.291 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-01T22:07:05.291 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:05.302 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-01T22:07:05.609 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 1 to deep-scrub 2023-12-01T22:07:05.620 INFO:tasks.ceph:Scrubbing osd.2 2023-12-01T22:07:05.621 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-01T22:07:05.770 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-01T22:07:05.770 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:05.782 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-01T22:07:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 2 to deep-scrub 2023-12-01T22:07:06.101 INFO:tasks.ceph:Scrubbing osd.3 2023-12-01T22:07:06.102 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-01T22:07:06.252 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:06.252 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-01T22:07:06.252 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:06.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-01T22:07:06.574 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 3 to deep-scrub 2023-12-01T22:07:06.586 INFO:tasks.ceph:Scrubbing osd.4 2023-12-01T22:07:06.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-01T22:07:06.736 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:06.737 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-01T22:07:06.737 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:06.748 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-01T22:07:07.057 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 4 to deep-scrub 2023-12-01T22:07:07.068 INFO:tasks.ceph:Scrubbing osd.5 2023-12-01T22:07:07.068 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-01T22:07:07.218 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-12-01T22:07:07.218 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-12-01T22:07:07.219 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-12-01T22:07:07.230 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-01T22:07:07.539 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 5 to deep-scrub 2023-12-01T22:07:07.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:07:07.886 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:07:07.899 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2023-12-01T22:07:07.371658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1791480,"kb_used_data":8904,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564439560,"statfs":{"total":579820584960,"available":577986109440,"internally_reserved":0,"allocated":9117696,"data_stored":4941552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629798+0000","last_change":"2023-12-01T22:06:13.501347+0000","last_active":"2023-12-01T22:06:14.629798+0000","last_peered":"2023-12-01T22:06:14.629798+0000","last_clean":"2023-12-01T22:06:14.629798+0000","last_became_active":"2023-12-01T22:06:13.501014+0000","last_became_peered":"2023-12-01T22:06:13.501014+0000","last_unstale":"2023-12-01T22:06:14.629798+0000","last_undegraded":"2023-12-01T22:06:14.629798+0000","last_fullsized":"2023-12-01T22:06:14.629798+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502631+0000","last_change":"2023-12-01T22:06:11.483134+0000","last_active":"2023-12-01T22:06:14.502631+0000","last_peered":"2023-12-01T22:06:14.502631+0000","last_clean":"2023-12-01T22:06:14.502631+0000","last_became_active":"2023-12-01T22:06:11.482843+0000","last_became_peered":"2023-12-01T22:06:11.482843+0000","last_unstale":"2023-12-01T22:06:14.502631+0000","last_undegraded":"2023-12-01T22:06:14.502631+0000","last_fullsized":"2023-12-01T22:06:14.502631+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629484+0000","last_change":"2023-12-01T22:06:13.501342+0000","last_active":"2023-12-01T22:06:14.629484+0000","last_peered":"2023-12-01T22:06:14.629484+0000","last_clean":"2023-12-01T22:06:14.629484+0000","last_became_active":"2023-12-01T22:06:13.500964+0000","last_became_peered":"2023-12-01T22:06:13.500964+0000","last_unstale":"2023-12-01T22:06:14.629484+0000","last_undegraded":"2023-12-01T22:06:14.629484+0000","last_fullsized":"2023-12-01T22:06:14.629484+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502610+0000","last_change":"2023-12-01T22:06:11.484990+0000","last_active":"2023-12-01T22:06:14.502610+0000","last_peered":"2023-12-01T22:06:14.502610+0000","last_clean":"2023-12-01T22:06:14.502610+0000","last_became_active":"2023-12-01T22:06:11.484646+0000","last_became_peered":"2023-12-01T22:06:11.484646+0000","last_unstale":"2023-12-01T22:06:14.502610+0000","last_undegraded":"2023-12-01T22:06:14.502610+0000","last_fullsized":"2023-12-01T22:06:14.502610+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502563+0000","last_change":"2023-12-01T22:06:13.503473+0000","last_active":"2023-12-01T22:06:14.502563+0000","last_peered":"2023-12-01T22:06:14.502563+0000","last_clean":"2023-12-01T22:06:14.502563+0000","last_became_active":"2023-12-01T22:06:13.503126+0000","last_became_peered":"2023-12-01T22:06:13.503126+0000","last_unstale":"2023-12-01T22:06:14.502563+0000","last_undegraded":"2023-12-01T22:06:14.502563+0000","last_fullsized":"2023-12-01T22:06:14.502563+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629795+0000","last_change":"2023-12-01T22:06:11.483341+0000","last_active":"2023-12-01T22:06:14.629795+0000","last_peered":"2023-12-01T22:06:14.629795+0000","last_clean":"2023-12-01T22:06:14.629795+0000","last_became_active":"2023-12-01T22:06:11.482990+0000","last_became_peered":"2023-12-01T22:06:11.482990+0000","last_unstale":"2023-12-01T22:06:14.629795+0000","last_undegraded":"2023-12-01T22:06:14.629795+0000","last_fullsized":"2023-12-01T22:06:14.629795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502999+0000","last_change":"2023-12-01T22:06:13.501802+0000","last_active":"2023-12-01T22:06:14.502999+0000","last_peered":"2023-12-01T22:06:14.502999+0000","last_clean":"2023-12-01T22:06:14.502999+0000","last_became_active":"2023-12-01T22:06:13.501277+0000","last_became_peered":"2023-12-01T22:06:13.501277+0000","last_unstale":"2023-12-01T22:06:14.502999+0000","last_undegraded":"2023-12-01T22:06:14.502999+0000","last_fullsized":"2023-12-01T22:06:14.502999+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629580+0000","last_change":"2023-12-01T22:06:11.484579+0000","last_active":"2023-12-01T22:06:14.629580+0000","last_peered":"2023-12-01T22:06:14.629580+0000","last_clean":"2023-12-01T22:06:14.629580+0000","last_became_active":"2023-12-01T22:06:11.484255+0000","last_became_peered":"2023-12-01T22:06:11.484255+0000","last_unstale":"2023-12-01T22:06:14.629580+0000","last_undegraded":"2023-12-01T22:06:14.629580+0000","last_fullsized":"2023-12-01T22:06:14.629580+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502458+0000","last_change":"2023-12-01T22:06:13.503682+0000","last_active":"2023-12-01T22:06:14.502458+0000","last_peered":"2023-12-01T22:06:14.502458+0000","last_clean":"2023-12-01T22:06:14.502458+0000","last_became_active":"2023-12-01T22:06:13.503327+0000","last_became_peered":"2023-12-01T22:06:13.503327+0000","last_unstale":"2023-12-01T22:06:14.502458+0000","last_undegraded":"2023-12-01T22:06:14.502458+0000","last_fullsized":"2023-12-01T22:06:14.502458+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502569+0000","last_change":"2023-12-01T22:06:11.482204+0000","last_active":"2023-12-01T22:06:14.502569+0000","last_peered":"2023-12-01T22:06:14.502569+0000","last_clean":"2023-12-01T22:06:14.502569+0000","last_became_active":"2023-12-01T22:06:11.481865+0000","last_became_peered":"2023-12-01T22:06:11.481865+0000","last_unstale":"2023-12-01T22:06:14.502569+0000","last_undegraded":"2023-12-01T22:06:14.502569+0000","last_fullsized":"2023-12-01T22:06:14.502569+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.628619+0000","last_change":"2023-12-01T22:06:13.500317+0000","last_active":"2023-12-01T22:06:14.628619+0000","last_peered":"2023-12-01T22:06:14.628619+0000","last_clean":"2023-12-01T22:06:14.628619+0000","last_became_active":"2023-12-01T22:06:13.500005+0000","last_became_peered":"2023-12-01T22:06:13.500005+0000","last_unstale":"2023-12-01T22:06:14.628619+0000","last_undegraded":"2023-12-01T22:06:14.628619+0000","last_fullsized":"2023-12-01T22:06:14.628619+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629650+0000","last_change":"2023-12-01T22:06:11.484953+0000","last_active":"2023-12-01T22:06:14.629650+0000","last_peered":"2023-12-01T22:06:14.629650+0000","last_clean":"2023-12-01T22:06:14.629650+0000","last_became_active":"2023-12-01T22:06:11.484596+0000","last_became_peered":"2023-12-01T22:06:11.484596+0000","last_unstale":"2023-12-01T22:06:14.629650+0000","last_undegraded":"2023-12-01T22:06:14.629650+0000","last_fullsized":"2023-12-01T22:06:14.629650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629536+0000","last_change":"2023-12-01T22:06:13.502547+0000","last_active":"2023-12-01T22:06:14.629536+0000","last_peered":"2023-12-01T22:06:14.629536+0000","last_clean":"2023-12-01T22:06:14.629536+0000","last_became_active":"2023-12-01T22:06:13.502190+0000","last_became_peered":"2023-12-01T22:06:13.502190+0000","last_unstale":"2023-12-01T22:06:14.629536+0000","last_undegraded":"2023-12-01T22:06:14.629536+0000","last_fullsized":"2023-12-01T22:06:14.629536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629855+0000","last_change":"2023-12-01T22:06:11.484374+0000","last_active":"2023-12-01T22:06:14.629855+0000","last_peered":"2023-12-01T22:06:14.629855+0000","last_clean":"2023-12-01T22:06:14.629855+0000","last_became_active":"2023-12-01T22:06:11.484068+0000","last_became_peered":"2023-12-01T22:06:11.484068+0000","last_unstale":"2023-12-01T22:06:14.629855+0000","last_undegraded":"2023-12-01T22:06:14.629855+0000","last_fullsized":"2023-12-01T22:06:14.629855+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629214+0000","last_change":"2023-12-01T22:06:13.503127+0000","last_active":"2023-12-01T22:06:14.629214+0000","last_peered":"2023-12-01T22:06:14.629214+0000","last_clean":"2023-12-01T22:06:14.629214+0000","last_became_active":"2023-12-01T22:06:13.502721+0000","last_became_peered":"2023-12-01T22:06:13.502721+0000","last_unstale":"2023-12-01T22:06:14.629214+0000","last_undegraded":"2023-12-01T22:06:14.629214+0000","last_fullsized":"2023-12-01T22:06:14.629214+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:11.268386+0000","last_clean_scrub_stamp":"2023-12-01T22:06:11.268386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629370+0000","last_change":"2023-12-01T22:06:11.486324+0000","last_active":"2023-12-01T22:06:14.629370+0000","last_peered":"2023-12-01T22:06:14.629370+0000","last_clean":"2023-12-01T22:06:14.629370+0000","last_became_active":"2023-12-01T22:06:11.486042+0000","last_became_peered":"2023-12-01T22:06:11.486042+0000","last_unstale":"2023-12-01T22:06:14.629370+0000","last_undegraded":"2023-12-01T22:06:14.629370+0000","last_fullsized":"2023-12-01T22:06:14.629370+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:06:10.202184+0000","last_clean_scrub_stamp":"2023-12-01T22:06:10.202184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629675+0000","last_change":"2023-12-01T22:06:04.970068+0000","last_active":"2023-12-01T22:06:14.629675+0000","last_peered":"2023-12-01T22:06:14.629675+0000","last_clean":"2023-12-01T22:06:14.629675+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.629675+0000","last_undegraded":"2023-12-01T22:06:14.629675+0000","last_fullsized":"2023-12-01T22:06:14.629675+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":153,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502444+0000","last_change":"2023-12-01T22:06:04.826303+0000","last_active":"2023-12-01T22:06:14.502444+0000","last_peered":"2023-12-01T22:06:14.502444+0000","last_clean":"2023-12-01T22:06:14.502444+0000","last_became_active":"2023-12-01T22:06:04.826034+0000","last_became_peered":"2023-12-01T22:06:04.826034+0000","last_unstale":"2023-12-01T22:06:14.502444+0000","last_undegraded":"2023-12-01T22:06:14.502444+0000","last_fullsized":"2023-12-01T22:06:14.502444+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629272+0000","last_change":"2023-12-01T22:06:04.826573+0000","last_active":"2023-12-01T22:06:14.629272+0000","last_peered":"2023-12-01T22:06:14.629272+0000","last_clean":"2023-12-01T22:06:14.629272+0000","last_became_active":"2023-12-01T22:06:04.826155+0000","last_became_peered":"2023-12-01T22:06:04.826155+0000","last_unstale":"2023-12-01T22:06:14.629272+0000","last_undegraded":"2023-12-01T22:06:14.629272+0000","last_fullsized":"2023-12-01T22:06:14.629272+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":151,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502541+0000","last_change":"2023-12-01T22:06:04.825578+0000","last_active":"2023-12-01T22:06:14.502541+0000","last_peered":"2023-12-01T22:06:14.502541+0000","last_clean":"2023-12-01T22:06:14.502541+0000","last_became_active":"2023-12-01T22:06:04.825295+0000","last_became_peered":"2023-12-01T22:06:04.825295+0000","last_unstale":"2023-12-01T22:06:14.502541+0000","last_undegraded":"2023-12-01T22:06:14.502541+0000","last_fullsized":"2023-12-01T22:06:14.502541+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502665+0000","last_change":"2023-12-01T22:06:04.969610+0000","last_active":"2023-12-01T22:06:14.502665+0000","last_peered":"2023-12-01T22:06:14.502665+0000","last_clean":"2023-12-01T22:06:14.502665+0000","last_became_active":"2023-12-01T22:06:04.969355+0000","last_became_peered":"2023-12-01T22:06:04.969355+0000","last_unstale":"2023-12-01T22:06:14.502665+0000","last_undegraded":"2023-12-01T22:06:14.502665+0000","last_fullsized":"2023-12-01T22:06:14.502665+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:52.351745+0000","last_clean_scrub_stamp":"2023-12-01T22:00:52.351745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502576+0000","last_change":"2023-12-01T22:06:04.825803+0000","last_active":"2023-12-01T22:06:14.502576+0000","last_peered":"2023-12-01T22:06:14.502576+0000","last_clean":"2023-12-01T22:06:14.502576+0000","last_became_active":"2023-12-01T22:06:04.825428+0000","last_became_peered":"2023-12-01T22:06:04.825428+0000","last_unstale":"2023-12-01T22:06:14.502576+0000","last_undegraded":"2023-12-01T22:06:14.502576+0000","last_fullsized":"2023-12-01T22:06:14.502576+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.502645+0000","last_change":"2023-12-01T22:06:04.969968+0000","last_active":"2023-12-01T22:06:14.502645+0000","last_peered":"2023-12-01T22:06:14.502645+0000","last_clean":"2023-12-01T22:06:14.502645+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:06:14.502645+0000","last_undegraded":"2023-12-01T22:06:14.502645+0000","last_fullsized":"2023-12-01T22:06:14.502645+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.629364+0000","last_change":"2023-12-01T22:06:04.826087+0000","last_active":"2023-12-01T22:06:14.629364+0000","last_peered":"2023-12-01T22:06:14.629364+0000","last_clean":"2023-12-01T22:06:14.629364+0000","last_became_active":"2023-12-01T22:06:04.825776+0000","last_became_peered":"2023-12-01T22:06:04.825776+0000","last_unstale":"2023-12-01T22:06:14.629364+0000","last_undegraded":"2023-12-01T22:06:14.629364+0000","last_fullsized":"2023-12-01T22:06:14.629364+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-12-01T22:06:14.503075+0000","last_change":"2023-12-01T22:06:04.824717+0000","last_active":"2023-12-01T22:06:14.503075+0000","last_peered":"2023-12-01T22:06:14.503075+0000","last_clean":"2023-12-01T22:06:14.503075+0000","last_became_active":"2023-12-01T22:06:04.824420+0000","last_became_peered":"2023-12-01T22:06:04.824420+0000","last_unstale":"2023-12-01T22:06:14.503075+0000","last_undegraded":"2023-12-01T22:06:14.503075+0000","last_fullsized":"2023-12-01T22:06:14.503075+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:00:54.365500+0000","last_clean_scrub_stamp":"2023-12-01T22:00:54.365500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.48799999999999999},"min":{"1min":0.28000000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.92000000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.73099999999999998},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.495},"min":{"1min":0.33300000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.77400000000000002,"15min":0.95399999999999996},"last":0.5},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68100000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49399999999999999},"min":{"1min":0.33900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.69499999999999995,"5min":0.73799999999999999,"15min":0.82999999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":0.73999999999999999,"5min":0.84099999999999997,"15min":0.872},"last":0.40999999999999998}]},{"osd":3,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.33800000000000002,"15min":0.34799999999999998},"min":{"1min":0.26500000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54000000000000004,"5min":0.56299999999999994,"15min":0.60999999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40600000000000003,"15min":0.374},"min":{"1min":0.34000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.67900000000000005},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.36099999999999999},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56899999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.374},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.40100000000000002},"min":{"1min":0.30499999999999999,"5min":0.248,"15min":0.247},"max":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.77600000000000002},"last":0.441}]}]},{"osd":4,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68100000000000005,"5min":0.746,"15min":0.746},"last":0.436},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48799999999999999,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.59599999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67400000000000004,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48399999999999999,"15min":0.495},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59899999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.499,"15min":0.47699999999999998},"min":{"1min":0.38400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64400000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.45900000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.33500000000000002},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46800000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Dec 1 22:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.245,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.39900000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.501,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57599999999999996}]}]},{"osd":3,"up_from":11,"seq":47244640334,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.496,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.61499999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51500000000000001,"15min":0.49299999999999999},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77600000000000002,"5min":0.877,"15min":0.877},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.495,"15min":0.47099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67200000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80800000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.59099999999999997,"15min":0.53200000000000003},"min":{"1min":0.28799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69199999999999995,"5min":8.4689999999999994,"15min":8.4689999999999994},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.60699999999999998,"15min":0.54600000000000004},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68300000000000005,"5min":7.819,"15min":7.819},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33600000000000002,"15min":0.35999999999999999},"min":{"1min":0.22700000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.45800000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.376},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.74399999999999999,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":0.39500000000000002}]},{"osd":5,"last update":"Fri Dec 1 22:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.42499999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41899999999999998,"15min":0.41499999999999998},"min":{"1min":0.28499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.42499999999999999,"5min":3.468,"15min":3.468},"last":0.43099999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.45100000000000001,"15min":0.372},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45000000000000001,"5min":12.333,"15min":12.333},"last":0.309},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.38700000000000001,"15min":0.36699999999999999},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53600000000000003,"5min":4.0609999999999999,"15min":4.0609999999999999},"last":0.29499999999999998}]},{"osd":1,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.49199999999999999,"15min":0.41099999999999998},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":0.57499999999999996,"5min":12.217000000000001,"15min":12.217000000000001},"last":0.378},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.55500000000000005,"15min":0.47199999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.5,"5min":13.824,"15min":13.824},"last":0.40200000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.83799999999999997,"15min":0.60899999999999999},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.68600000000000005,"5min":29.079999999999998,"15min":29.079999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.70699999999999996,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29499999999999998},"max":{"1min":0.81599999999999995,"5min":16.920999999999999,"15min":16.920999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54800000000000004,"15min":0.50600000000000001},"min":{"1min":0.35199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70299999999999996,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.436},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57699999999999996,"15min":0.53700000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83299999999999996,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.50800000000000001}]},{"osd":5,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57099999999999995,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.32700000000000001,"15min":0.314},"max":{"1min":0.753,"5min":4.0149999999999997,"15min":4.0149999999999997},"last":0.442},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":2.524,"15min":2.524},"last":0.48499999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640333,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.48199999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.246},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36399999999999999,"15min":0.373},"min":{"1min":0.24199999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51700000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.27600000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.57199999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38600000000000001,"15min":0.38},"min":{"1min":0.29999999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56000000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.30099999999999999}]},{"osd":3,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76600000000000001,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.503},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69499999999999995,"5min":0.751,"15min":0.751},"last":0.47199999999999998}]},{"osd":4,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.50900000000000001},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.64800000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.748,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.502}]},{"osd":5,"last update":"Fri Dec 1 22:06:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.41699999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76300000000000001,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.70999999999999996,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48899999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640335,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.33900000000000002,"15min":0.34200000000000003},"min":{"1min":0.23300000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.51300000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.371,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.30399999999999999}]},{"osd":2,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.58799999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39700000000000002,"15min":0.41399999999999998},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67200000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32400000000000001}]},{"osd":3,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79000000000000004,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57199999999999995}]},{"osd":4,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.318,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Dec 1 22:06:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:07:07.903 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.904 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.904 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.904 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.904 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.904 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.904 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.905 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.905 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.905 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.905 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.905 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.905 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.906 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.906 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2023-12-01T22:06:11.268386+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=11, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.906 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2023-12-01T22:06:10.202184+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=6, tm_sec=10, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.906 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.906 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.906 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.907 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.907 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-01T22:00:52.351745+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=52, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.907 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.907 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.907 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.908 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-01T22:00:54.365500+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=0, tm_sec=54, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=22, tm_min=7, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T22:07:07.908 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-01T22:07:27.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T22:07:28.227 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-12-01T22:07:28.228 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-12-01T22:07:28.240 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2023-12-01T22:07:27.376410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1791480,"kb_used_data":8904,"kb_used_omap":0,"kb_used_meta":1782528,"kb_avail":564439560,"statfs":{"total":579820584960,"available":577986109440,"internally_reserved":0,"allocated":9117696,"data_stored":4941552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1825308672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:10.569218+0000","last_change":"2023-12-01T22:07:10.569218+0000","last_active":"2023-12-01T22:07:10.569218+0000","last_peered":"2023-12-01T22:07:10.569218+0000","last_clean":"2023-12-01T22:07:10.569218+0000","last_became_active":"2023-12-01T22:06:13.501014+0000","last_became_peered":"2023-12-01T22:06:13.501014+0000","last_unstale":"2023-12-01T22:07:10.569218+0000","last_undegraded":"2023-12-01T22:07:10.569218+0000","last_fullsized":"2023-12-01T22:07:10.569218+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:10.569118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:10.569118+0000","last_clean_scrub_stamp":"2023-12-01T22:07:10.569118+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:14.805900+0000","last_change":"2023-12-01T22:07:14.805900+0000","last_active":"2023-12-01T22:07:14.805900+0000","last_peered":"2023-12-01T22:07:14.805900+0000","last_clean":"2023-12-01T22:07:14.805900+0000","last_became_active":"2023-12-01T22:06:11.482843+0000","last_became_peered":"2023-12-01T22:06:11.482843+0000","last_unstale":"2023-12-01T22:07:14.805900+0000","last_undegraded":"2023-12-01T22:07:14.805900+0000","last_fullsized":"2023-12-01T22:07:14.805900+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:14.805815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:14.805815+0000","last_clean_scrub_stamp":"2023-12-01T22:07:14.805815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:14.462040+0000","last_change":"2023-12-01T22:07:14.462040+0000","last_active":"2023-12-01T22:07:14.462040+0000","last_peered":"2023-12-01T22:07:14.462040+0000","last_clean":"2023-12-01T22:07:14.462040+0000","last_became_active":"2023-12-01T22:06:13.500964+0000","last_became_peered":"2023-12-01T22:06:13.500964+0000","last_unstale":"2023-12-01T22:07:14.462040+0000","last_undegraded":"2023-12-01T22:07:14.462040+0000","last_fullsized":"2023-12-01T22:07:14.462040+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:14.461934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:14.461934+0000","last_clean_scrub_stamp":"2023-12-01T22:07:14.461934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:13.835290+0000","last_change":"2023-12-01T22:07:13.835290+0000","last_active":"2023-12-01T22:07:13.835290+0000","last_peered":"2023-12-01T22:07:13.835290+0000","last_clean":"2023-12-01T22:07:13.835290+0000","last_became_active":"2023-12-01T22:06:11.484646+0000","last_became_peered":"2023-12-01T22:06:11.484646+0000","last_unstale":"2023-12-01T22:07:13.835290+0000","last_undegraded":"2023-12-01T22:07:13.835290+0000","last_fullsized":"2023-12-01T22:07:13.835290+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:13.835187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:13.835187+0000","last_clean_scrub_stamp":"2023-12-01T22:07:13.835187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:18.709182+0000","last_change":"2023-12-01T22:07:18.709182+0000","last_active":"2023-12-01T22:07:18.709182+0000","last_peered":"2023-12-01T22:07:18.709182+0000","last_clean":"2023-12-01T22:07:18.709182+0000","last_became_active":"2023-12-01T22:06:13.503126+0000","last_became_peered":"2023-12-01T22:06:13.503126+0000","last_unstale":"2023-12-01T22:07:18.709182+0000","last_undegraded":"2023-12-01T22:07:18.709182+0000","last_fullsized":"2023-12-01T22:07:18.709182+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:18.709070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:18.709070+0000","last_clean_scrub_stamp":"2023-12-01T22:07:18.709070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:08.549922+0000","last_change":"2023-12-01T22:07:08.549922+0000","last_active":"2023-12-01T22:07:08.549922+0000","last_peered":"2023-12-01T22:07:08.549922+0000","last_clean":"2023-12-01T22:07:08.549922+0000","last_became_active":"2023-12-01T22:06:11.482990+0000","last_became_peered":"2023-12-01T22:06:11.482990+0000","last_unstale":"2023-12-01T22:07:08.549922+0000","last_undegraded":"2023-12-01T22:07:08.549922+0000","last_fullsized":"2023-12-01T22:07:08.549922+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:08.549840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:08.549840+0000","last_clean_scrub_stamp":"2023-12-01T22:07:08.549840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:12.663987+0000","last_change":"2023-12-01T22:07:12.663987+0000","last_active":"2023-12-01T22:07:12.663987+0000","last_peered":"2023-12-01T22:07:12.663987+0000","last_clean":"2023-12-01T22:07:12.663987+0000","last_became_active":"2023-12-01T22:06:13.501277+0000","last_became_peered":"2023-12-01T22:06:13.501277+0000","last_unstale":"2023-12-01T22:07:12.663987+0000","last_undegraded":"2023-12-01T22:07:12.663987+0000","last_fullsized":"2023-12-01T22:07:12.663987+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:12.663879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:12.663879+0000","last_clean_scrub_stamp":"2023-12-01T22:07:12.663879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:19.442201+0000","last_change":"2023-12-01T22:07:19.442201+0000","last_active":"2023-12-01T22:07:19.442201+0000","last_peered":"2023-12-01T22:07:19.442201+0000","last_clean":"2023-12-01T22:07:19.442201+0000","last_became_active":"2023-12-01T22:06:11.484255+0000","last_became_peered":"2023-12-01T22:06:11.484255+0000","last_unstale":"2023-12-01T22:07:19.442201+0000","last_undegraded":"2023-12-01T22:07:19.442201+0000","last_fullsized":"2023-12-01T22:07:19.442201+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:19.442111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:19.442111+0000","last_clean_scrub_stamp":"2023-12-01T22:07:19.442111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:16.733258+0000","last_change":"2023-12-01T22:07:16.733258+0000","last_active":"2023-12-01T22:07:16.733258+0000","last_peered":"2023-12-01T22:07:16.733258+0000","last_clean":"2023-12-01T22:07:16.733258+0000","last_became_active":"2023-12-01T22:06:13.503327+0000","last_became_peered":"2023-12-01T22:06:13.503327+0000","last_unstale":"2023-12-01T22:07:16.733258+0000","last_undegraded":"2023-12-01T22:07:16.733258+0000","last_fullsized":"2023-12-01T22:07:16.733258+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:16.733163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:16.733163+0000","last_clean_scrub_stamp":"2023-12-01T22:07:16.733163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:11.702868+0000","last_change":"2023-12-01T22:07:11.702868+0000","last_active":"2023-12-01T22:07:11.702868+0000","last_peered":"2023-12-01T22:07:11.702868+0000","last_clean":"2023-12-01T22:07:11.702868+0000","last_became_active":"2023-12-01T22:06:11.481865+0000","last_became_peered":"2023-12-01T22:06:11.481865+0000","last_unstale":"2023-12-01T22:07:11.702868+0000","last_undegraded":"2023-12-01T22:07:11.702868+0000","last_fullsized":"2023-12-01T22:07:11.702868+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:11.702770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:11.702770+0000","last_clean_scrub_stamp":"2023-12-01T22:07:11.702770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:11.947753+0000","last_change":"2023-12-01T22:07:11.947753+0000","last_active":"2023-12-01T22:07:11.947753+0000","last_peered":"2023-12-01T22:07:11.947753+0000","last_clean":"2023-12-01T22:07:11.947753+0000","last_became_active":"2023-12-01T22:06:13.500005+0000","last_became_peered":"2023-12-01T22:06:13.500005+0000","last_unstale":"2023-12-01T22:07:11.947753+0000","last_undegraded":"2023-12-01T22:07:11.947753+0000","last_fullsized":"2023-12-01T22:07:11.947753+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:11.947650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:11.947650+0000","last_clean_scrub_stamp":"2023-12-01T22:07:11.947650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:18.403984+0000","last_change":"2023-12-01T22:07:18.403984+0000","last_active":"2023-12-01T22:07:18.403984+0000","last_peered":"2023-12-01T22:07:18.403984+0000","last_clean":"2023-12-01T22:07:18.403984+0000","last_became_active":"2023-12-01T22:06:11.484596+0000","last_became_peered":"2023-12-01T22:06:11.484596+0000","last_unstale":"2023-12-01T22:07:18.403984+0000","last_undegraded":"2023-12-01T22:07:18.403984+0000","last_fullsized":"2023-12-01T22:07:18.403984+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:18.403873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:18.403873+0000","last_clean_scrub_stamp":"2023-12-01T22:07:18.403873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:20.483408+0000","last_change":"2023-12-01T22:07:20.483408+0000","last_active":"2023-12-01T22:07:20.483408+0000","last_peered":"2023-12-01T22:07:20.483408+0000","last_clean":"2023-12-01T22:07:20.483408+0000","last_became_active":"2023-12-01T22:06:13.502190+0000","last_became_peered":"2023-12-01T22:06:13.502190+0000","last_unstale":"2023-12-01T22:07:20.483408+0000","last_undegraded":"2023-12-01T22:07:20.483408+0000","last_fullsized":"2023-12-01T22:07:20.483408+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:20.483296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:20.483296+0000","last_clean_scrub_stamp":"2023-12-01T22:07:20.483296+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:06.518585+0000","last_change":"2023-12-01T22:07:06.518585+0000","last_active":"2023-12-01T22:07:06.518585+0000","last_peered":"2023-12-01T22:07:06.518585+0000","last_clean":"2023-12-01T22:07:06.518585+0000","last_became_active":"2023-12-01T22:06:11.484068+0000","last_became_peered":"2023-12-01T22:06:11.484068+0000","last_unstale":"2023-12-01T22:07:06.518585+0000","last_undegraded":"2023-12-01T22:07:06.518585+0000","last_fullsized":"2023-12-01T22:07:06.518585+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:06.518514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:06.518514+0000","last_clean_scrub_stamp":"2023-12-01T22:07:06.518514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:08.447661+0000","last_change":"2023-12-01T22:07:08.447661+0000","last_active":"2023-12-01T22:07:08.447661+0000","last_peered":"2023-12-01T22:07:08.447661+0000","last_clean":"2023-12-01T22:07:08.447661+0000","last_became_active":"2023-12-01T22:06:13.502721+0000","last_became_peered":"2023-12-01T22:06:13.502721+0000","last_unstale":"2023-12-01T22:07:08.447661+0000","last_undegraded":"2023-12-01T22:07:08.447661+0000","last_fullsized":"2023-12-01T22:07:08.447661+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:08.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:08.447566+0000","last_clean_scrub_stamp":"2023-12-01T22:07:08.447566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:14.434806+0000","last_change":"2023-12-01T22:07:14.434806+0000","last_active":"2023-12-01T22:07:14.434806+0000","last_peered":"2023-12-01T22:07:14.434806+0000","last_clean":"2023-12-01T22:07:14.434806+0000","last_became_active":"2023-12-01T22:06:11.486042+0000","last_became_peered":"2023-12-01T22:06:11.486042+0000","last_unstale":"2023-12-01T22:07:14.434806+0000","last_undegraded":"2023-12-01T22:07:14.434806+0000","last_fullsized":"2023-12-01T22:07:14.434806+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:14.434704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:14.434704+0000","last_clean_scrub_stamp":"2023-12-01T22:07:14.434704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":140,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:10.320221+0000","last_change":"2023-12-01T22:07:10.320221+0000","last_active":"2023-12-01T22:07:10.320221+0000","last_peered":"2023-12-01T22:07:10.320221+0000","last_clean":"2023-12-01T22:07:10.320221+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:07:10.320221+0000","last_undegraded":"2023-12-01T22:07:10.320221+0000","last_fullsized":"2023-12-01T22:07:10.320221+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:10.320116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:10.320116+0000","last_clean_scrub_stamp":"2023-12-01T22:07:10.320116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":158,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:08.727037+0000","last_change":"2023-12-01T22:07:08.727037+0000","last_active":"2023-12-01T22:07:08.727037+0000","last_peered":"2023-12-01T22:07:08.727037+0000","last_clean":"2023-12-01T22:07:08.727037+0000","last_became_active":"2023-12-01T22:06:04.826034+0000","last_became_peered":"2023-12-01T22:06:04.826034+0000","last_unstale":"2023-12-01T22:07:08.727037+0000","last_undegraded":"2023-12-01T22:07:08.727037+0000","last_fullsized":"2023-12-01T22:07:08.727037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:08.726907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:08.726907+0000","last_clean_scrub_stamp":"2023-12-01T22:07:08.726907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":138,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:07.409816+0000","last_change":"2023-12-01T22:07:07.409816+0000","last_active":"2023-12-01T22:07:07.409816+0000","last_peered":"2023-12-01T22:07:07.409816+0000","last_clean":"2023-12-01T22:07:07.409816+0000","last_became_active":"2023-12-01T22:06:04.826155+0000","last_became_peered":"2023-12-01T22:06:04.826155+0000","last_unstale":"2023-12-01T22:07:07.409816+0000","last_undegraded":"2023-12-01T22:07:07.409816+0000","last_fullsized":"2023-12-01T22:07:07.409816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:07.409689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:07.409689+0000","last_clean_scrub_stamp":"2023-12-01T22:07:07.409689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":156,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:10.730018+0000","last_change":"2023-12-01T22:07:10.730018+0000","last_active":"2023-12-01T22:07:10.730018+0000","last_peered":"2023-12-01T22:07:10.730018+0000","last_clean":"2023-12-01T22:07:10.730018+0000","last_became_active":"2023-12-01T22:06:04.825295+0000","last_became_peered":"2023-12-01T22:06:04.825295+0000","last_unstale":"2023-12-01T22:07:10.730018+0000","last_undegraded":"2023-12-01T22:07:10.730018+0000","last_fullsized":"2023-12-01T22:07:10.730018+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:10.729892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:10.729892+0000","last_clean_scrub_stamp":"2023-12-01T22:07:10.729892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":146,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:06.687031+0000","last_change":"2023-12-01T22:07:06.687031+0000","last_active":"2023-12-01T22:07:06.687031+0000","last_peered":"2023-12-01T22:07:06.687031+0000","last_clean":"2023-12-01T22:07:06.687031+0000","last_became_active":"2023-12-01T22:06:04.969355+0000","last_became_peered":"2023-12-01T22:06:04.969355+0000","last_unstale":"2023-12-01T22:07:06.687031+0000","last_undegraded":"2023-12-01T22:07:06.687031+0000","last_fullsized":"2023-12-01T22:07:06.687031+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:06.686931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:06.686931+0000","last_clean_scrub_stamp":"2023-12-01T22:07:06.686931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:08.730994+0000","last_change":"2023-12-01T22:07:08.730994+0000","last_active":"2023-12-01T22:07:08.730994+0000","last_peered":"2023-12-01T22:07:08.730994+0000","last_clean":"2023-12-01T22:07:08.730994+0000","last_became_active":"2023-12-01T22:06:04.825428+0000","last_became_peered":"2023-12-01T22:06:04.825428+0000","last_unstale":"2023-12-01T22:07:08.730994+0000","last_undegraded":"2023-12-01T22:07:08.730994+0000","last_fullsized":"2023-12-01T22:07:08.730994+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:08.730884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:08.730884+0000","last_clean_scrub_stamp":"2023-12-01T22:07:08.730884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:09.721369+0000","last_change":"2023-12-01T22:07:09.721369+0000","last_active":"2023-12-01T22:07:09.721369+0000","last_peered":"2023-12-01T22:07:09.721369+0000","last_clean":"2023-12-01T22:07:09.721369+0000","last_became_active":"2023-12-01T22:06:04.969751+0000","last_became_peered":"2023-12-01T22:06:04.969751+0000","last_unstale":"2023-12-01T22:07:09.721369+0000","last_undegraded":"2023-12-01T22:07:09.721369+0000","last_fullsized":"2023-12-01T22:07:09.721369+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:09.721265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:09.721265+0000","last_clean_scrub_stamp":"2023-12-01T22:07:09.721265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":139,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:12.363743+0000","last_change":"2023-12-01T22:07:12.363743+0000","last_active":"2023-12-01T22:07:12.363743+0000","last_peered":"2023-12-01T22:07:12.363743+0000","last_clean":"2023-12-01T22:07:12.363743+0000","last_became_active":"2023-12-01T22:06:04.825776+0000","last_became_peered":"2023-12-01T22:06:04.825776+0000","last_unstale":"2023-12-01T22:07:12.363743+0000","last_undegraded":"2023-12-01T22:07:12.363743+0000","last_fullsized":"2023-12-01T22:07:12.363743+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:12.363647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:12.363647+0000","last_clean_scrub_stamp":"2023-12-01T22:07:12.363647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-12-01T22:07:10.724111+0000","last_change":"2023-12-01T22:07:10.724111+0000","last_active":"2023-12-01T22:07:10.724111+0000","last_peered":"2023-12-01T22:07:10.724111+0000","last_clean":"2023-12-01T22:07:10.724111+0000","last_became_active":"2023-12-01T22:06:04.824420+0000","last_became_peered":"2023-12-01T22:06:04.824420+0000","last_unstale":"2023-12-01T22:07:10.724111+0000","last_undegraded":"2023-12-01T22:07:10.724111+0000","last_fullsized":"2023-12-01T22:07:10.724111+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T22:07:10.724010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T22:07:10.724010+0000","last_clean_scrub_stamp":"2023-12-01T22:07:10.724010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47099999999999997,"15min":0.48799999999999999},"min":{"1min":0.28000000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.64000000000000001,"5min":0.68899999999999995,"15min":0.92000000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46899999999999997,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64500000000000002,"5min":0.72899999999999998,"15min":0.73099999999999998},"last":0.28599999999999998}]},{"osd":1,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.495},"min":{"1min":0.33300000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.77400000000000002,"15min":0.95399999999999996},"last":0.376},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68100000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.36799999999999999}]},{"osd":2,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48699999999999999,"15min":0.49399999999999999},"min":{"1min":0.33900000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.69499999999999995,"5min":0.73799999999999999,"15min":0.82999999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.5},"min":{"1min":0.38,"5min":0.32000000000000001,"15min":0.30599999999999999},"max":{"1min":0.73999999999999999,"5min":0.84099999999999997,"15min":0.872},"last":0.35299999999999998}]},{"osd":3,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.33800000000000002,"15min":0.34799999999999998},"min":{"1min":0.26500000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54000000000000004,"5min":0.56299999999999994,"15min":0.60999999999999999},"last":0.312},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40600000000000003,"15min":0.374},"min":{"1min":0.34000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.67900000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Dec 1 22:07:00 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.36099999999999999},"min":{"1min":0.25600000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56899999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.40100000000000002},"min":{"1min":0.30499999999999999,"5min":0.248,"15min":0.247},"max":{"1min":0.60199999999999998,"5min":0.67000000000000004,"15min":0.77600000000000002},"last":0.38700000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.46500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66000000000000003,"5min":0.746,"15min":0.746},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48699999999999999,"15min":0.47999999999999998},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Dec 1 22:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.48899999999999999,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65100000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.501},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51200000000000001,"15min":0.51700000000000002},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71699999999999997,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.442}]},{"osd":2,"last update":"Fri Dec 1 22:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47899999999999998,"15min":0.49299999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69999999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.47899999999999998},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.52600000000000002}]},{"osd":3,"last update":"Fri Dec 1 22:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.34899999999999998,"15min":0.34200000000000003},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.34999999999999998,"15min":0.34000000000000002},"min":{"1min":0.29099999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42999999999999999}]},{"osd":5,"last update":"Fri Dec 1 22:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.377,"15min":0.35499999999999998},"min":{"1min":0.27000000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.67800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.40500000000000003},"min":{"1min":0.30099999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81899999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55800000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640338,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.501,"15min":0.46899999999999997},"min":{"1min":0.29499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66800000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.51200000000000001,"15min":0.49199999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63300000000000001,"5min":0.877,"15min":0.877},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.34000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64200000000000002,"5min":0.76300000000000001,"15min":0.79100000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51000000000000001,"15min":0.46999999999999997},"min":{"1min":0.35599999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.60099999999999998,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.58399999999999996,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70899999999999996,"5min":8.4689999999999994,"15min":8.4689999999999994},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.60199999999999998,"15min":0.54400000000000004},"min":{"1min":0.374,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69899999999999995,"5min":7.819,"15min":7.819},"last":0.55700000000000005}]},{"osd":4,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.35999999999999999},"min":{"1min":0.245,"5min":0.216,"15min":0.216},"max":{"1min":0.46400000000000002,"5min":0.55900000000000005,"15min":0.57899999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.317,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47499999999999998,"5min":4.1980000000000004,"15min":4.1980000000000004},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39300000000000002,"15min":0.39400000000000002},"min":{"1min":0.312,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47399999999999998,"5min":3.4239999999999999,"15min":3.4239999999999999},"last":0.437},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.372,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":3.468,"15min":3.468},"last":0.53300000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.45100000000000001,"15min":0.372},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.45000000000000001,"5min":12.333,"15min":12.333},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.38700000000000001,"15min":0.36699999999999999},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53600000000000003,"5min":4.0609999999999999,"15min":4.0609999999999999},"last":0.46000000000000002}]},{"osd":1,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.49199999999999999,"15min":0.41099999999999998},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.22800000000000001},"max":{"1min":0.57499999999999996,"5min":12.217000000000001,"15min":12.217000000000001},"last":0.502},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.55500000000000005,"15min":0.47199999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.5,"5min":13.824,"15min":13.824},"last":0.627}]},{"osd":3,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.83799999999999997,"15min":0.60899999999999999},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.68600000000000005,"5min":29.079999999999998,"15min":29.079999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.70699999999999996,"15min":0.57399999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29499999999999998},"max":{"1min":0.81599999999999995,"5min":16.920999999999999,"15min":16.920999999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54800000000000004,"15min":0.50600000000000001},"min":{"1min":0.35199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70299999999999996,"5min":2.8999999999999999,"15min":2.8999999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57699999999999996,"15min":0.53700000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.83299999999999996,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.57399999999999995}]},{"osd":5,"last update":"Fri Dec 1 22:07:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.57099999999999995,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.32700000000000001,"15min":0.314},"max":{"1min":0.753,"5min":4.0149999999999997,"15min":4.0149999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79600000000000004,"5min":2.524,"15min":2.524},"last":0.65500000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640337,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 22:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56599999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.371},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52800000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Dec 1 22:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.65400000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.30099999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Dec 1 22:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53600000000000003,"15min":0.504},"min":{"1min":0.35999999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66000000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52900000000000003,"15min":0.505},"min":{"1min":0.38800000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.622}]},{"osd":4,"last update":"Fri Dec 1 22:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53000000000000003,"15min":0.51000000000000001},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.755,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.58299999999999996}]},{"osd":5,"last update":"Fri Dec 1 22:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68200000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.73899999999999999,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64500000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640339,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298580,"kb_used_data":1484,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94073260,"statfs":{"total":96636764160,"available":96331018240,"internally_reserved":0,"allocated":1519616,"data_stored":823592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.33400000000000002,"15min":0.34000000000000002},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.45300000000000001,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.36399999999999999,"15min":0.36599999999999999},"min":{"1min":0.26700000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.47799999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.32600000000000001}]},{"osd":2,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35699999999999998,"15min":0.36499999999999999},"min":{"1min":0.26500000000000001,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":0.53400000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.38200000000000001,"15min":0.40899999999999997},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.502,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.373}]},{"osd":3,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.36499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.51200000000000001},"min":{"1min":0.34100000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.79000000000000004,"15min":0.88200000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.70799999999999996,"5min":0.751,"15min":0.84099999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68500000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.59999999999999998}]},{"osd":5,"last update":"Fri Dec 1 22:07:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":0.67000000000000004,"5min":0.80700000000000005,"15min":0.82699999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.54800000000000004},"min":{"1min":0.40799999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68700000000000006,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T22:07:28.242 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-01T22:07:28.577 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-01T22:07:28.577 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-01T22:07:28.577 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-01T22:07:28.577 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.621 INFO:tasks.ceph.osd.0:Stopped 2023-12-01T22:07:28.622 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-01T22:07:28.622 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.660 INFO:tasks.ceph.osd.1:Stopped 2023-12-01T22:07:28.660 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-01T22:07:28.660 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.728 INFO:tasks.ceph.osd.2:Stopped 2023-12-01T22:07:28.728 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-01T22:07:28.728 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.759 INFO:tasks.ceph.osd.3:Stopped 2023-12-01T22:07:28.759 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-01T22:07:28.759 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.837 INFO:tasks.ceph.osd.4:Stopped 2023-12-01T22:07:28.838 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-01T22:07:28.838 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.907 INFO:tasks.ceph.osd.5:Stopped 2023-12-01T22:07:28.908 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-01T22:07:28.908 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-01T22:07:28.908 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.974 INFO:tasks.ceph.mgr.x:Stopped 2023-12-01T22:07:28.974 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-01T22:07:28.974 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-01T22:07:28.975 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:28.986 INFO:tasks.ceph.mon.a:Stopped 2023-12-01T22:07:28.986 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-01T22:07:28.986 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:29.038 INFO:tasks.ceph.mon.c:Stopped 2023-12-01T22:07:29.038 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-01T22:07:29.038 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T22:07:29.050 INFO:tasks.ceph.mon.b:Stopped 2023-12-01T22:07:29.050 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-01T22:07:29.051 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-12-01T22:07:29.107 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:07:29.108 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-01T22:07:29.331 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:07:29.332 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-01T22:07:29.438 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:07:29.441 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-01T22:07:29.513 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:07:29.513 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-01T22:07:29.652 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:07:29.652 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-01T22:07:29.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:07:29.749 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-01T22:07:29.823 INFO:tasks.ceph:Archiving mon data... 2023-12-01T22:07:29.824 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/data/mon.a.tgz 2023-12-01T22:07:29.824 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-12-01T22:07:29.839 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.gLGojJZ0bz 2023-12-01T22:07:29.839 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.gLGojJZ0bz 2023-12-01T22:07:30.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.gLGojJZ0bz 2023-12-01T22:07:30.209 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.gLGojJZ0bz is 963KB 2023-12-01T22:07:30.313 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.gLGojJZ0bz 2023-12-01T22:07:30.328 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/data/mon.c.tgz 2023-12-01T22:07:30.328 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-12-01T22:07:30.382 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.mcMMlqERZ2 2023-12-01T22:07:30.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.mcMMlqERZ2 2023-12-01T22:07:30.635 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.mcMMlqERZ2 2023-12-01T22:07:30.717 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.mcMMlqERZ2 is 965KB 2023-12-01T22:07:30.821 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.mcMMlqERZ2 2023-12-01T22:07:30.836 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/data/mon.b.tgz 2023-12-01T22:07:30.836 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-12-01T22:07:30.852 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.DQk3D7WV4M 2023-12-01T22:07:30.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DQk3D7WV4M 2023-12-01T22:07:31.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.DQk3D7WV4M 2023-12-01T22:07:31.195 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.DQk3D7WV4M is 964KB 2023-12-01T22:07:31.299 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.DQk3D7WV4M 2023-12-01T22:07:31.313 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-01T22:07:31.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-01T22:07:31.316 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-01T22:07:31.377 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-01T22:07:31.377 DEBUG:teuthology.orchestra.run.smithi039:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-01T22:07:31.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-01T22:07:31.407 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-01T22:07:31.441 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-01T22:07:31.447 INFO:tasks.ceph:Archiving crash dumps... 2023-12-01T22:07:31.448 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/remote/smithi039/crash 2023-12-01T22:07:31.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-01T22:07:31.478 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/remote/smithi170/crash 2023-12-01T22:07:31.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-01T22:07:31.510 INFO:tasks.ceph:Compressing logs... 2023-12-01T22:07:31.511 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-01T22:07:31.523 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-01T22:07:31.615 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91980.log 2023-12-01T22:07:31.615 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91356.log 2023-12-01T22:07:31.615 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91850.log 2023-12-01T22:07:31.615 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91980.log.gz 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86770.log 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91356.log.gz 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92472.log 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91850.log.gz 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.86770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92149.log 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86770.log.gz 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92472.log.gz 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94612.log 2023-12-01T22:07:31.616 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90317.log 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92149.log.gz 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92682.log 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94612.log.gz 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92106.log 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90317.log.gz 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95292.log 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92682.log.gz 2023-12-01T22:07:31.617 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92106.log.gz 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94528.log 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93407.log 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95292.log.gz 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93744.log 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94528.log.gz 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92388.log 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93407.log.gz 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91482.log 2023-12-01T22:07:31.618 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93744.log.gz 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91766.log 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92388.log.gz 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93105.log 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91482.log.gz 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92304.log 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91766.log.gz 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95922.log 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93105.log.gz 2023-12-01T22:07:31.619 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93828.log 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92304.log.gz 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89926.log 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95922.log.gz 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92022.log 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93828.log.gz 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94780.log 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89926.log.gz 2023-12-01T22:07:31.620 INFO:teuthology.orchestra.run.smithi039.stderr:gzip /var/log/ceph/ceph-client.admin.92022.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.93618.log 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92022.log.gz 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93962.log 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94780.log.gz 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93786.log 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93618.log.gz 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90527.log 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93962.log.gz 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91060.log 2023-12-01T22:07:31.621 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93786.log.gz 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89800.log 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90527.log.gz 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.91060.log: -- /var/log/ceph/ceph-client.admin.90695.log 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91060.log.gz 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89800.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89800.log.gz 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91892.log 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90695.log.gz 2023-12-01T22:07:31.622 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90359.log 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.91892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93660.log 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91892.log.gz 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86459.log 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90359.log.gz 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91938.log 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93660.log.gz 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.86459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94276.log 2023-12-01T22:07:31.623 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86459.log.gz 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89506.log 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91938.log.gz 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91724.log 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94276.log.gz 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89842.log 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89506.log.gz 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92598.log 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91724.log.gz 2023-12-01T22:07:31.624 INFO:teuthology.orchestra.run.smithi039.stderr: 92.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92850.log 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89842.log.gz 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92430.log 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92598.log.gz 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89548.log 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92850.log.gz 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89632.log 2023-12-01T22:07:31.625 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92430.log.gz 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94444.log 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89548.log.gz 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90233.log 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89632.log.gz 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94486.log 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94444.log.gz 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90611.log 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90233.log.gz 2023-12-01T22:07:31.626 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92192.log 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94486.log.gz 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90485.log 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90611.log.gz 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92192.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92192.log.gz 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93702.log 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90485.log.gz 2023-12-01T22:07:31.627 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92640.log 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95838.log 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93702.log.gz 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93920.log 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92640.log.gz 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95418.log 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95838.log.gz 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91188.log 2023-12-01T22:07:31.628 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93920.log.gz 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91314.log 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95418.log.gz 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89884.log 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91188.log.gz 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90653.log 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91314.log.gz 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94231.log 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89884.log.gz 2023-12-01T22:07:31.629 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95796.log 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90653.log.gz 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94231.log.gz 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92766.log 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89758.log 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95796.log.gz 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92346.log 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-01T22:07:31.630 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92766.log.gz 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89758.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.89464.log 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89758.log.gz 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92724.log 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92346.log.gz 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90569.log 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89464.log.gz 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90850.log 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92724.log.gz 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91272.log 2023-12-01T22:07:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90569.log.gz 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94004.log 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90850.log.gz 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93449.log 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91272.log.gz 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94654.log 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94004.log.gz 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95502.log 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93449.log.gz 2023-12-01T22:07:31.632 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93365.log 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94654.log.gz 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92937.log 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95502.log.gz 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89674.log 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93365.log.gz 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89296.log 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92937.log.gz 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94830.log 2023-12-01T22:07:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89674.log.gz 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89590.log 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89296.log.gz 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94570.log 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94830.log.gz 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89380.log 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89590.log.gz 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95334.log 2023-12-01T22:07:31.634 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94570.log.gz 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93197.log 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89380.log.gz 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95544.log 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95334.log.gz 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94696.log 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93197.log.gz 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92979.log 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95544.log.gz 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90136.log 2023-12-01T22:07:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94696.log.gz 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90275.log 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92979.log.gz 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90136.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90136.log.gz 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82081.log 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90275.log.gz 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95754.log 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.tmp-client.admin.82081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95376.log 2023-12-01T22:07:31.636 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.82081.log.gz 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95628.log 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95754.log.gz 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94998.log 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95376.log.gz 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95628.log.gz 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91598.log 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94872.log 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94998.log.gz 2023-12-01T22:07:31.637 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91808.log 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91598.log.gz 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94872.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94872.log.gz 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95880.log 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91808.log.gz 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93063.log 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90781.log 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95880.log.gz 2023-12-01T22:07:31.638 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92064.log 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93063.log.gz 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90781.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90781.log.gz 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92808.log 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92064.log.gz 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92514.log 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95460.log 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92808.log.gz 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94738.log 2023-12-01T22:07:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92514.log.gz 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93870.log 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95460.log.gz 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90052.log 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94738.log.gz 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93576.log 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93870.log.gz 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90401.log 2023-12-01T22:07:31.640 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90052.log.gz 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91682.log 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93576.log.gz 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91146.log 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90401.log: 24.4% -- replaced with /var/log/ceph/ceph-client.admin.90401.log.gz 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95124.log 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91682.log.gz 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91440.log 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91146.log.gz 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93281.log 2023-12-01T22:07:31.641 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95124.log.gz 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90443.log 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91440.log.gz 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95250.log 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93281.log.gz 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94318.log 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90443.log.gz 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91018.log 2023-12-01T22:07:31.642 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95250.log.gz 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95082.log 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94318.log.gz 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92895.log 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91018.log.gz 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89968.log 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95082.log.gz 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95040.log 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92895.log.gz 2023-12-01T22:07:31.643 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89968.log.gz 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91230.log 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95040.log.gz 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86686.log 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91398.log 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91230.log.gz 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr:gzip /var/log/ceph/ceph-client.admin.86686.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.89241.log 2023-12-01T22:07:31.644 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86686.log.gz 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96006.log 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91398.log.gz 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89422.log 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89241.log.gz 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.96006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90976.log 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96006.log.gz 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94402.log 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89422.log.gz 2023-12-01T22:07:31.645 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90737.log 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90976.log.gz 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95670.log 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94402.log.gz 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94148.log 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90737.log.gz 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93323.log 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95670.log.gz 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92556.log 2023-12-01T22:07:31.646 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94148.log.gz 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:gzip /var/log/ceph/ceph-client.admin.93323.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95208.log 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93323.log.gz 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.92556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92556.log.gz 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95208.log.gz 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89716.log 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93147.log 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90191.log 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89716.log.gz 2023-12-01T22:07:31.647 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93492.log 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93147.log.gz 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90934.log 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90191.log.gz 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94046.log 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93492.log.gz 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93239.log 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90934.log.gz 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95964.log 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94046.log.gz 2023-12-01T22:07:31.648 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89338.log 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93239.log.gz 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95712.log 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95964.log.gz 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.89338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90094.log 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89338.log.gz 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94956.log 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95712.log.gz 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94914.log 2023-12-01T22:07:31.649 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90094.log.gz 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95166.log 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94956.log.gz 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91524.log 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94914.log.gz 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93534.log 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95166.log.gz 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93021.log 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91524.log.gz 2023-12-01T22:07:31.650 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90010.log 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93534.log.gz 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.93021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96046.log 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93021.log.gz 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.90010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94360.log 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90010.log.gz 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.96046.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96046.log.gz 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95586.log 2023-12-01T22:07:31.651 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94360.log.gz 2023-12-01T22:07:31.652 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91102.log 2023-12-01T22:07:31.652 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.95586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94088.log 2023-12-01T22:07:31.652 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95586.log.gz 2023-12-01T22:07:31.652 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.91102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91102.log.gz 2023-12-01T22:07:31.652 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.94088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94088.log.gz 2023-12-01T22:07:31.696 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92793.log 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90878.log 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90904.log 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92793.log.gz 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93496.log 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90878.log.gz 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93080.log 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90904.log.gz 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93621.log 2023-12-01T22:07:31.697 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93496.log.gz 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91135.log 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93080.log.gz 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91846.log 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93621.log.gz 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90851.log 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91135.log.gz 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93411.log 2023-12-01T22:07:31.698 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91846.log.gz 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93005.log 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90851.log.gz 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89986.log 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93411.log.gz 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90390.log 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93005.log.gz 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93694.log 2023-12-01T22:07:31.699 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89986.log.gz 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91721.log 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90390.log.gz 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92768.log 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93694.log.gz 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89647.log 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91721.log.gz 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91296.log 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92768.log.gz 2023-12-01T22:07:31.700 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92136.log 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89647.log.gz 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90145.log 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91296.log.gz 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91187.log 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92136.log.gz 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91498.log 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90145.log.gz 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90293.log 2023-12-01T22:07:31.701 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91187.log.gz 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92360.log 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91498.log.gz 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93442.log 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90293.log.gz 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93207.log 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92360.log.gz 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92643.log 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93442.log.gz 2023-12-01T22:07:31.702 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93571.log 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93207.log.gz 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91959.log 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92643.log.gz 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92312.log 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93571.log.gz 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90440.log 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91959.log.gz 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90983.log 2023-12-01T22:07:31.703 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92312.log.gz 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90203.log 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90440.log.gz 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92562.log 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90983.log.gz 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr:gzip /var/log/ceph/ceph-client.admin.90203.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.89599.log 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90203.log.gz 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92562.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92562.log.gz 2023-12-01T22:07:31.704 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91350.log 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89599.log.gz 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86407.log 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92013.log 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91350.log.gz 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92113.log 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86407.log.gz 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90650.log 2023-12-01T22:07:31.705 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92013.log.gz 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93232.log 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92113.log.gz 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90249.log 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90650.log.gz 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91473.log 2023-12-01T22:07:31.706 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93232.log.gz 2023-12-01T22:07:31.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86283.log 2023-12-01T22:07:31.709 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90249.log.gz 2023-12-01T22:07:31.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92743.log 2023-12-01T22:07:31.709 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91473.log.gz 2023-12-01T22:07:31.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91546.log 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86283.log.gz 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90675.log 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92743.log.gz 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93055.log 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91546.log.gz 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89842.log 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90675.log.gz 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93796.log 2023-12-01T22:07:31.710 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93055.log.gz 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86497.log 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89842.log.gz 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91161.log 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93796.log.gz 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90315.log 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86497.log.gz 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93282.log 2023-12-01T22:07:31.711 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91161.log.gz 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89959.log 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90315.log.gz 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92036.log 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93282.log.gz 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91246.log 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89959.log.gz 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91821.log 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92036.log.gz 2023-12-01T22:07:31.712 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89792.log 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91246.log.gz 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93596.log 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91821.log.gz 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91646.log 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89792.log.gz 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93596.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93596.log.gz 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89720.log 2023-12-01T22:07:31.713 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91646.log.gz 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92536.log 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91425.log 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89720.log.gz 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93671.log 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92536.log.gz 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92459.log 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91425.log.gz 2023-12-01T22:07:31.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89672.log 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93671.log.gz 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90550.log 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92459.log.gz 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93359.log 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89672.log.gz 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93257.log 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90550.log.gz 2023-12-01T22:07:31.715 INFO:teuthology.orchestra.run.smithi170.stderr: 91.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90775.log 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93359.log.gz 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93257.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93257.log.gz 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93107.log 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90775.log.gz 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92383.log 2023-12-01T22:07:31.716 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93107.log.gz 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91271.log 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89430.log 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92383.log.gz 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91271.log.gz 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89624.log 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91084.log 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89430.log.gz 2023-12-01T22:07:31.717 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89624.log.gz 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91008.log 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92184.log 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91084.log.gz 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91008.log.gz 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91213.log 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89817.log 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92184.log.gz 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91213.log.gzgzip 2023-12-01T22:07:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.86456.log 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89928.log 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89817.log.gz 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90750.log 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86456.log.gz 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91746.log 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89928.log.gz 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92236.log 2023-12-01T22:07:31.719 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90750.log.gz 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86365.log 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91746.log.gz 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92818.log 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92236.log.gz 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93872.log 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86365.log.gz 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89870.log 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92818.log.gz 2023-12-01T22:07:31.720 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93182.log 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93872.log.gz 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92408.log 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89870.log.gz 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92511.log 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93182.log.gz 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92408.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.92408.log.gz-- 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90956.log 2023-12-01T22:07:31.721 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93821.log 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92511.log.gz 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90956.log.gz 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90600.log 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89372.log 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93821.log.gz 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90600.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.90600.log.gz/var/log/ceph/ceph-client.admin.91671.log 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90700.log 2023-12-01T22:07:31.722 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89372.log.gz 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91671.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.91671.log.gz 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.92337.log 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90525.log 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90700.log.gz 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92955.log 0.0% 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92337.log.gz 2023-12-01T22:07:31.723 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93521.log 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90525.log.gz 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93646.log 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.92955.log.gz 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91796.log 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93521.log.gz 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91521.log 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93646.log.gz 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91796.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.92433.log -- replaced with /var/log/ceph/ceph-client.admin.91796.log.gz 2023-12-01T22:07:31.724 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86324.log 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91521.log.gz 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92433.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92433.log.gz 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.86324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90063.log 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86324.log.gz 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91109.log 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92485.log 2023-12-01T22:07:31.725 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90063.log.gz 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89456.log 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91109.log.gz 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89553.log 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92485.log.gz 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90725.log 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89456.log.gz 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93030.log 2023-12-01T22:07:31.726 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89553.log.gz 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91034.log 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90725.log.gz 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92693.log 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93030.log.gz 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89397.log 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91034.log.gz 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92061.log 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92693.log.gz 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89507.log 2023-12-01T22:07:31.727 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89397.log.gz 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92061.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.92061.log.gz/var/log/ceph/ceph-client.admin.92620.log 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91771.log 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89507.log.gz 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90415.log 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92620.log.gz 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91771.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.91771.log.gz-5 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:--verbose -- /var/log/ceph/ceph-client.admin.90040.log 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90088.log 2023-12-01T22:07:31.728 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90415.log.gz 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93546.log 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90040.log.gz 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90088.log.gz 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93469.log 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93721.log 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93546.log.gz 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90931.log 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93469.log.gz 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89481.log 2023-12-01T22:07:31.729 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93721.log.gz 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92980.log 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90931.log.gz 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93846.log 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89481.log.gz 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90271.log 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92980.log.gz 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90825.log 2023-12-01T22:07:31.730 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93846.log.gz 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89742.log 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90271.log.gz 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90825.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.90825.log.gz 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:-- /var/log/ceph/ceph-client.admin.91596.log 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93132.log 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89742.log.gz 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90120.log 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91596.log.gz 2023-12-01T22:07:31.731 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90226.log 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93132.log.gz 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91696.log 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90120.log.gz 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93332.log 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90226.log.gz 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92086.log 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91696.log.gz 2023-12-01T22:07:31.732 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93332.log.gz 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93386.log 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92086.log.gz 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92668.log 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90365.log 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93386.log.gz 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91450.log 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92668.log.gz 2023-12-01T22:07:31.733 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91932.log 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90365.log.gz 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92211.log 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91450.log.gz 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91323.log 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91932.log.gz 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92843.log 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92211.log.gz 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92718.log 2023-12-01T22:07:31.734 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91323.log.gz 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90575.log 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92843.log.gz 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90017.log 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92718.log.gz 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92928.log 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90575.log.gz 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90465.log 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90017.log.gz 2023-12-01T22:07:31.735 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91986.log 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92928.log.gz 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90465.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90465.log.gz 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93307.log 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91986.log.gz 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91400.log 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93307.log.gz 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91400.log.gz 2023-12-01T22:07:31.736 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89530.log 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92587.log 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90800.log 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89530.log.gz 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92289.log 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92587.log.gz 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92262.log 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90800.log.gz 2023-12-01T22:07:31.737 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93773.log 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92289.log.gz 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93748.log 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92262.log.gz 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93773.log.gz 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93748.log.gz 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93155.log 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91571.log 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.93155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93155.log.gz 2023-12-01T22:07:31.738 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91571.log.gz 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91059.log 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89576.log 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90625.log 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91059.log.gz 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90340.log 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89576.log.gz 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90625.log.gz 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90340.log.gz 2023-12-01T22:07:31.739 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91621.log 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89695.log 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91621.log.gz 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89695.log.gz 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92159.log 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90177.log 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.92159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89767.log 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92159.log.gz 2023-12-01T22:07:31.740 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.90177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89901.log 2023-12-01T22:07:31.741 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90177.log.gz 2023-12-01T22:07:31.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89767.log.gz 2023-12-01T22:07:31.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.89901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89901.log.gz 2023-12-01T22:07:31.741 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91375.log 2023-12-01T22:07:31.741 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.91375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91375.log.gz 2023-12-01T22:07:31.741 INFO:teuthology.orchestra.run.smithi170.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-01T22:07:31.742 INFO:teuthology.orchestra.run.smithi039.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-01T22:07:31.830 INFO:teuthology.orchestra.run.smithi170.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-01T22:07:31.888 INFO:teuthology.orchestra.run.smithi039.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-01T22:07:32.239 INFO:teuthology.orchestra.run.smithi039.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-01T22:07:32.240 INFO:teuthology.orchestra.run.smithi170.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-01T22:07:32.255 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-01T22:07:32.277 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-01T22:07:32.279 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-01T22:07:32.279 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m0.707s 2023-12-01T22:07:32.279 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m2.197s 2023-12-01T22:07:32.279 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.159s 2023-12-01T22:07:32.304 INFO:teuthology.orchestra.run.smithi039.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-01T22:07:32.311 INFO:teuthology.orchestra.run.smithi039.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-01T22:07:32.313 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-12-01T22:07:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m0.778s 2023-12-01T22:07:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m2.562s 2023-12-01T22:07:32.313 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.153s 2023-12-01T22:07:32.313 INFO:tasks.ceph:Archiving logs... 2023-12-01T22:07:32.314 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/remote/smithi039/log 2023-12-01T22:07:32.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-01T22:07:32.875 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/remote/smithi170/log 2023-12-01T22:07:32.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-01T22:07:33.360 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-01T22:07:33.374 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-01T22:07:33.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-01T22:07:33.377 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-01T22:07:33.453 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-01T22:07:33.464 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-01T22:07:33.464 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T22:07:33.467 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T22:07:33.480 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-12-01T22:07:33.481 INFO:teuthology.orchestra.run.smithi170.stderr:bash: ntpq: command not found 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:210 Number of sources = 4 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 -474us[ -474us] +/- 76ms 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -979us[ -971us] +/- 62ms 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv03.front.sepia.ceph.com 3 6 377 50 +1692us[+1699us] +/- 74ms 2023-12-01T22:07:33.483 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:07:33.485 INFO:teuthology.orchestra.run.smithi170.stdout:210 Number of sources = 4 2023-12-01T22:07:33.485 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T22:07:33.485 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-12-01T22:07:33.485 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 -485us[ -507us] +/- 76ms 2023-12-01T22:07:33.485 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -984us[ -984us] +/- 62ms 2023-12-01T22:07:33.486 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv03.front.sepia.ceph.com 3 6 377 50 +1695us[+1672us] +/- 74ms 2023-12-01T22:07:33.486 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T22:07:33.486 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-01T22:07:33.496 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-01T22:07:33.496 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-01T22:07:33.505 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-01T22:07:33.622 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-01T22:07:33.711 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T22:07:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:07:33.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T22:07:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:07:33.773 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-01T22:07:33.784 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-01T22:07:33.794 INFO:teuthology.task.internal:Duration was 1212.335545 seconds 2023-12-01T22:07:33.795 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-01T22:07:33.805 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-01T22:07:33.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-01T22:07:33.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-01T22:07:33.851 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T22:07:33.856 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T22:07:34.197 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-01T22:07:34.198 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2023-12-01T22:07:34.198 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-01T22:07:34.218 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-12-01T22:07:34.219 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-01T22:07:34.237 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-01T22:07:34.238 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-01T22:07:34.261 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-01T22:07:34.316 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-01T22:07:34.316 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-01T22:07:34.341 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-01T22:07:34.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-01T22:07:34.436 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-01T22:07:34.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-01T22:07:34.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-01T22:07:34.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-01T22:07:34.496 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:07:34.505 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:07:34.526 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2023-12-01T22:07:34.550 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-12-01T22:07:34.564 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:07:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:07:34.595 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-01T22:07:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:07:34.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-01T22:07:34.630 INFO:teuthology.task.internal:Transferring archived files... 2023-12-01T22:07:34.631 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/remote/smithi039 2023-12-01T22:07:34.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-01T22:07:34.674 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306/remote/smithi170 2023-12-01T22:07:34.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-01T22:07:34.709 INFO:teuthology.task.internal:Removing archive directory... 2023-12-01T22:07:34.709 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-01T22:07:34.711 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-01T22:07:34.758 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-01T22:07:34.769 INFO:teuthology.task.internal:Not uploading archives. 2023-12-01T22:07:34.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-01T22:07:34.785 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-01T22:07:34.785 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-01T22:07:34.788 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-01T22:07:34.801 INFO:teuthology.orchestra.run.smithi039.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 1 22:07 /home/ubuntu/cephtest 2023-12-01T22:07:34.812 INFO:teuthology.orchestra.run.smithi170.stdout: 262153 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 1 22:07 /home/ubuntu/cephtest 2023-12-01T22:07:34.812 INFO:teuthology.orchestra.run.smithi170.stdout: 661082 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 1 22:01 /home/ubuntu/cephtest/mnt.0 2023-12-01T22:07:34.812 INFO:teuthology.orchestra.run.smithi170.stdout: 661083 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Dec 1 22:03 /home/ubuntu/cephtest/mnt.0/client.0 2023-12-01T22:07:34.813 INFO:teuthology.orchestra.run.smithi170.stdout: 661123 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 1 22:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-01T22:07:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T22:07:34.814 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-01T22:07:34.814 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-01T22:07:34.822 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-01T22:07:34.833 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-01T22:07:34.845 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi170 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-12-01T22:07:34.845 INFO:teuthology.nuke:Checking targets against current locks 2023-12-01T22:07:34.876 DEBUG:teuthology.nuke:shortname: smithi039 2023-12-01T22:07:34.876 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T22:07:34.902 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:38:31.002543', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHXlX9zAwNcMwQQU3DB/EAeYCGpSjW2Nux9nhT2vXGdTFPf5SeLso4wUcOT9/x/mg0OnwCREdKDZcPGAiTVcXiY='} 2023-12-01T22:07:34.906 DEBUG:teuthology.nuke:shortname: smithi170 2023-12-01T22:07:34.906 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T22:07:34.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_16:53:58-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7474306', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 21:38:31.003982', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWQdVma+lyjNWkZyjNVt3uVkytJ8p1wyCBD+94MlywamkEMtBb61G4EUuRHQxCXZHEFEPB8KciAtaYAKz6p/mk='} 2023-12-01T22:07:34.961 INFO:teuthology.orchestra.console.smithi039:Power off 2023-12-01T22:07:34.962 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T22:07:34.999 INFO:teuthology.orchestra.console.smithi170:Power off 2023-12-01T22:07:35.000 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T22:07:35.010 DEBUG:teuthology.orchestra.console.smithi039:power off output: Chassis Power Control: Down/Off 2023-12-01T22:07:35.011 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:07:35.023 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-12-01T22:07:35.024 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:07:35.035 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2023-12-01T22:07:35.049 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-01T22:07:39.036 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:07:39.050 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:07:39.148 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2023-12-01T22:07:39.161 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-01T22:07:43.150 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:07:43.162 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T22:07:43.261 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is off 2023-12-01T22:07:43.272 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-12-01T22:07:43.362 INFO:teuthology.orchestra.console.smithi039:Power off completed 2023-12-01T22:07:43.373 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-12-01T22:07:43.479 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} duration: 1212.3355450630188 failure_reason: 'Command failed (workunit test mon/crush_ops.sh) on smithi170 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ae640b47502d4af0a4472ba651a67d69 status: fail success: false 2023-12-01T22:07:43.479 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T22:07:43.562 INFO:teuthology.run:FAIL